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Abstract 


The  goal  of  this  study  was  to  analyze  the  effectiveness  of  the  new  SWOS-at- 
Sea  training  for  newly  commissioned  surface  warfare  officers  that  was  introduced  in 
2003.  The  new  regime  combined  self-paced  computer-based  training  (CBT)  with 
on-the-job  training  (OJT)  on-board  an  officer’s  ship.  The  study  relied  on  a  variety  of 
analytical  techniques,  including  a  literature  review  of  CBT  and  OJT  training, 
interviews  and  focus  groups  with  junior  and  senior  surface  warfare  officers,  and 
statistical  analysis  of  test  scores  at  the  Surface  Officer  Division  Officer  Course 
(SWOSDOC).  The  literature  review  pointed  out  that  no  previous  studies  had 
analyzed  a  learning  course  that  involved  CBT  when  the  student  was  also  performing 
a  full-time  job.  Nonetheless,  the  literature  review  noted  that  structured  OJT  is 
preferred  to  unstructured  OJT,  as  in  SWOS-at-Sea.  Interviews  also  indicated  that 
division  officers  prefer  face-to-face  learning  and  they  felt  they  were  trying  to  perform 
as  division  officers  without  the  requisite  skills.  The  statistical  analysis  of  test  scores 
at  the  SWOSDOC  ‘leveling’  course  found  significant  differences  in  performance 
between  ship  type,  ship  home  port  location,  commissioning  source,  undergraduate 
major,  and  gender.  The  results  highlighted  significant  differences  in  the  learning 
environments  aboard  ships  and  suggested  the  need  for  additional  research  on 
training  opportunities  offered  aboard  ships. 

Keywords:  Surface  Warfare,  Officers,  training 
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Executive  Summary 


For  many  years,  newly  commissioned  Surface  Warfare  Officers  (SWOs)  in 
the  Navy  were  trained  at  the  Surface  Warfare  Officer  School  Division  Officer  Course 
(SWOSDOC),  after  which  they  reported  to  their  first  ship  assignment.  In  January 
2003,  that  training  was  replaced  by  what  is  called  SWOS-at-Sea  Training,  which 
combines  computer-based  training  (CBT)  with  on-the-job  training  (OJT)  conducted 
onboard  an  officer’s  first  ship.  Once  the  Officer  of  the  Deck  Underway  (OOD-U/W) 
qualification  is  achieved,  officers  report  to  SWOSDOC  for  a  3-week  course  of 
instruction  for  “leveling”  (i.e.,  to  bring  everyone  to  the  same  level  of  knowledge). 
Feedback  suggests  that  the  new  program  may  not  be  working  as  well  as  intended. 

The  purpose  of  this  research  was  to  identify:  (1 )  the  parameters  of  effective 
CBT  and  OJT  and  how  those  compare  to  the  implementation  of  the  SWOS-at-Sea 
training;  (2)  how  stakeholders  perceive  the  effectiveness  of  the  SWOS-at-Sea 
training;  (3)  how  effectively  junior  officers  are  learning  from  CBT;  (4)  what 
measurable  factors  (e.g.,  gender,  commissioning  source,  and  ship  type)  are 
predictors  of  training  effectiveness;  and  (5)  the  extent  to  which  leveling  actually 
occurs  by  the  end  of  the  3-week  SWOSDOC  course  of  instruction.  The  study  uses 
several  methods  of  analysis:  literature  was  reviewed;  interviews  and  focus  groups 
were  conducted;  and  data  on  performance  on  a  pretest  (administered  by 
SWOSDOC  to  12  classes,  a  total  of  733  students)  was  analyzed. 

The  literature  review  shows  that  the  Navy,  like  other  organizations,  has  a 
history  of  converting  traditional  classroom  training  to  various  forms  of  mediated 
instruction,  including  CBT,  to  reduce  costs  and  provide  learning  at  any  time  and  any 
place.  Yet,  many  prior  studies  have  questioned  whether  using  CBT  is  associated 
with  a  reduction  in  the  quality  of  instruction.  The  multitude  of  factors  that  affect 
quality  and,  in  many  cases,  inadequate  metrics  make  it  difficult  for  analysts  to  make 
definitive  statements  about  the  transfer  of  learning  that  one  can  expect  from  CBT. 
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For  the  most  part,  students  prefer  face-to-face  learning  to  mediated 
instruction,  with  “blended  solutions”  (a  combination  of  the  two)  as  a  second  choice. 
The  parameters  of  effective  CBT,  as  identified  in  the  literature,  were  not  present 
consistently  in  the  SWOS-at-Sea  training.  For  example,  interactivity,  collaboration, 
and  a  supportive  learning  community  are  related  to  higher  satisfaction  and 
achievement,  but  interviews  with  SWOs  showed  that  they  did  not  find  the  CBT 
engaging,  interactive,  or  interesting.  “Death  by  PowerPoint”  was  a  common 
complaint. 

Many  of  the  elements  in  the  original  design  of  the  SWOS-at-Sea  training  were 
not  implemented  as  intended.  For  example,  there  was  inconsistency  in  mentoring 
and  in  accountability  for  those  completing  the  CBT.  Where  ships’  personnel  were 
involved  in  the  training  through  mentoring  and  other  programs  (e.g.,  weekly 
meetings  at  “SWO  University”  with  the  Commanding  Officer),  there  was  greater 
satisfaction.  An  interesting  finding  was  that,  contrary  to  expectations  based  on  what 
the  researchers  know  about  Generation  Y,  the  group  interviewed  did  not  like 
learning  from  a  computer.  They  told  us  that  computers  are  for  work  and  games  but 
not  learning — they  still  want  human  contact. 

It  is  particularly  noteworthy  that  no  studies  were  found  that  describe  the 
effects  of  a  distance  learning  course  like  the  SWOS  CBT  that  is  mandated  in 
addition  to  a  student’s  full-time  job,  as  opposed  to  during  work  hours  or  in  a  full-time 
learning  mode. 

The  review  of  literature  on  OJT  shows  that  many  organizations  depend  on 
OJT  for  training  to  acquire  or  maintain  skills,  to  reduce  costs,  and  to  effectively 
transfer  training  to  job  performance.  The  literature  is  quite  clear  on  what  constitutes 
effective  OJT.  In  particular,  structured  (planned  and  systematic)  OJT  is  preferred  to 
unstructured  OJT  because  it  produces  consistent  training  that  ensures  that  training 
objectives  are  achieved.  The  SWOS  at-Sea  training  represents  unstructured  OJT. 
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A  last,  but  key,  complaint  heard  in  interviews  was  that  the  students  felt 
frustrated,  if  not  embarrassed,  to  be  put  in  the  position  of  running  a  division  without 
the  requisite  skills  and  knowledge  to  do  the  job.  Many  felt  that  the  training  sends  the 
message  that  they  are  not  valued  to  the  same  extent  as  other  unrestricted  line 
officers,  and  this  may  affect  their  decision  to  stay  in  the  Navy.  Findings  from  the 
interviews  for  background  and  with  six  post-command  Navy  Captains  confirmed 
what  we  heard  from  the  students  at  SWOSDOC. 

With  respect  to  the  analysis  of  the  quantitative  data  on  test  scores  at 
SWOSDOC,  the  general  success  of  self-directed  CBT  paired  with  OJT  (as 
measured  by  the  pre-test  administered  at  SWOSDOC)  is  found  to  vary  significantly 
with  respect  to  ship  type  and  home  port  assignment.  In  addition,  test  scores  varied 
with  pre-commissioning  factors  such  as  commissioning  source,  quality  of 
undergraduate  college,  and  academic  majors,  as  well  as  demographic 
characteristics — especially  gender.  The  findings  of  the  statistical  analysis  of  test 
scores  are  summarized  as  follows; 

1 .  Officers  assigned  to  the  primary  line  ships  (cruisers,  destroyers,  and 
frigates,  called  CRUDES)  are  significantly  less  likely  to  pass  the  CBT 
module  tests. 

2.  Those  assigned  to  home  ports  on  the  Atlantic  coast  have  significantly 
lower  passing  rates  on  the  CBT  tests  than  do  others. 

3.  Naval  Academy  graduates  had  significantly  lower  passing  rates  than 
otherwise  similar  officers  commissioned  by  NROTC  and  OCS. 

4.  The  quality  of  one’s  undergraduate  college — used  to  approximate 
academic  ability — is  directly  related  to  performance  on  the  CBT 
modules. 

5.  Graduates  with  technical  majors  have  higher  passing  rates  on  the  CBT 
tests  than  those  with  non-technical  skills.  However,  it  is  important  to 
keep  in  mind  that  the  CBT  modules  do  not  capture  all  of  the  skills 
needed  by  junior  officers  to  become  successful  Division  Officers 
(DIVOs). 

6.  Women  have  significantly  lower  pass  rates  than  men. 

7.  Few  differences  are  found  by  ethnicity — with  the  exception  of  the 
scores  on  the  Conning  Officer  Virtual  Environment  exam  given  toward 
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the  end  of  SWOSDOC;  all  minorities  are  less  likely  to  pass  this  exam 
as  compared  to  their  majority  counterparts. 

8.  Most,  if  not  all,  of  the  test  score  differences  disappear  when  junior 
officers  are  given  introductory  training  in  a  traditional  classroom 
environment.  In  other  words,  leveling  occurs  by  the  end  of 
SWOSDOC. 

In  summary,  the  quantitative  analysis  in  this  study  addresses  personal 
achievement  on  initial  junior  officer  tests  and  exams,  which  are  related  to  the 
technical  skill  competency  of  Division  Officers.  The  researchers  uncovered 
achievement  differences  on  the  tests  at  SWOSDOC  that  are  related  to  pre¬ 
commissioning  factors  as  well  as  to  the  environments  aboard  ships  and  home  ports. 
Such  variations  may  possibly  be  indicative  of  a  larger  issue  related  to  the  early 
training  opportunities  of  junior  officers  in  the  surface  navy.  To  the  extent  this  is  true, 
it  is  in  the  best  interest  of  the  community  to  investigate  in  more  depth  why  training 
environments  for  junior  officers  differ  to  the  degree  that  has  been  found  in  this  study. 

The  study  recommends  that  future  studies  investigate  the  source  of 
differences  in  test  score  performance  at  SWOSDOC  based  on  ship  type  and  home 
port.  In  addition,  test  score  differences  based  on  academic  background  (technical 
degree)  and  commissioning  source  are  worthy  of  further  analysis.  Finally,  it  is 
especially  important  that  future  research  be  directed  at  determining  why  females  and 
minorities  appear  to  perform  worse  on  the  initial  tests  at  SWOSDOC  leveling  school. 
It  is  highly  recommended  that  additional  resources  be  devoted  to  a  better 
understanding  of  learning  and  training  opportunities  aboard  ships  before  decisions 
are  made  regarding  how  training  will  be  divided  between  classroom  and  self-paced 
CBT  aboard  ships. 
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Introduction 


In  January  2003,  the  Navy  implemented  a  new  training  program  for  the 
Surface  Warfare  Officer’s  School  Division  Officer  Course  (SWOSDOC).  Prior  to 
January  2003,  newly  commissioned  officers  reported  to  the  Surface  Warfare  Officer 
School  (SWOS)  in  Newport,  Rhode  Island,  to  complete  SWOSDOC.  The  initial 
training  course  equipped  junior  officers  with  the  fundamental  knowledge  necessary 
to  function  as  a  division  officer  (DIVO)  and  to  qualify  as  a  Surface  Warfare  Officer 
(SWO).  The  format  consisted  of  six  months  of  classroom-style  training.  Upon 
graduation,  the  officer  reported  to  his  or  her  ship  to  complete  the  initial  SWO 
qualification  process  through  further  on-the-job  training.  The  entire  qualification 
process  took  approximately  24  months  from  the  beginning  of  SWOSDOC  to  final 
qualification  by  the  ship’s  Commanding  Officer  (CO). 

Under  the  new  training  process,  newly  commissioned  officers  report  directly 
to  their  ships  and  individually  complete  the  training  onboard  the  ship.^  The  new 
training  program  consists  of  two  distinct  curricula.  The  Navy  repackaged  the 
classroom-style  training  of  SWOSDOC  into  a  computer-based  training  (CBT)  format 
(SWOS-at-Sea  2.0),  combined  with  on-the-job  training  (OJT)  collectively  referred  to 
as  SWOS-at-Sea  training.  Upon  completion  of  the  SWOS-at-Sea  training  and  of 
qualifying  as  Officer-of-the-Deck  Underway  (OOD  U/W),  junior  officers  report  to  the 
Surface  Warfare  Officer’s  School  (SWOS),  where  they  complete  the  second 
curriculum — a  3-week  resident  course  that  provides  advanced  proficiency  training 
designed  to  complement  the  at-sea  program  of  study. 


^  The  literature  uses  the  following  terms  synonymously:  Computer-based  Training  (CBT),  Computer- 
based  Instruction  (CBI),  Computer-assisted-Training  (CAT)  and  Computer-assisted  Instruction  (CAI). 
This  study  uses  CBT  unless  quoting  from  a  study  that  uses  one  of  the  synonymous  terms.  In  that 
case,  the  appropriate  term  is  used.  Further,  there  is  a  distinction  between  CBT  and  E-learning 
(sometimes  called  Web-learning),  as  the  latter  specifically  takes  place  over  a  network  or  internet 
connection.  Nevertheless,  E-learning  and  CBT  are  both  computer-based. 
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The  Navy’s  reasons  for  changing  the  training  program  were  to  speed  the  time 
necessary  to  achieve  the  Surface  Warfare  Qualification  and  to  reduce  overall 
training  costs  of  surface  warfare  officers.  As  predicted,  the  time  to  qualify  as  a 
surface  officer  has  fallen  from  roughly  24  to  16  months  (Rowden,  2008;  Surface 
Warfare  Enterprise,  2007).  Nonetheless,  some  observers  began  to  doubt  the 
efficacy  of  sending  all  newly  commissioned  officers  to  sea  without  first  giving  them 
better  training  in  the  fundamentals  of  being  a  division  officer.  This  concern  was 
summarized  by  the  comments  of  a  junior  officer  in  a  US  Naval  Institute  on-line 
publication: 

What  NETC  may  fail  to  recognize,  however,  is  the  quality  of  the  division 
officer  they’re  now  producing — one  who  lacks  baseline  knowledge  when  first 
reporting  onboard  and  must  struggle  to  qualify  through  a  process  of  trial,  error 
and  failure,  while  being  forced  to  sit  in  front  of  a  computer  for  hours  on  end. 
The  fact  is,  the  current  training  program  is  ineffective  and  wasteful. 

Shipboard  life  is  not  conducive  to  completing  a  program  like  the  “SWOS-in-a- 
Box,”  and  the  follow-on  3-week  finishing  school  is,  quite  simply,  a  paid 
vacation.  (Shovlin,  2008,  p.  1) 

The  primary  goal  of  this  research  is  to  investigate  how  well  the  re-engineered 
SWOS-at-Sea  training  is  preparing  junior  officers  for  Surface  Warfare  Officer  (SWO) 
Qualification  and  their  jobs  as  division  officers.  The  research  will  examine  how  well 
junior  officer  SWO  candidates  are  learning  from  the  computer-based  approach.  The 
research  is  grounded  on  the  overall  goal  of  Navy  SWO  division  officer  training, 
namely  to  effectively  prepare  prospective  junior  officers  for  SWO  qualification. 

The  study  attempts  to  answer  several  questions; 


1 .  What  does  a  review  of  the  literature  tell  us  about  the  parameters  of 
effective  CBT  and  OJT,  and  how  the  actual  implementation  of  the 
SWOS-at-Sea  training  compare  to  the  ideal  program? 

2.  How  do  stakeholders  perceive  the  effectiveness  of  the  SWOS-at-Sea 
training? 

3.  How  effectively  are  junior  officers  learning  from  CBT?  SWOS 
administers  a  pretest  derived  from  the  SWOS-at-Sea  training  to 
students  arriving  at  the  SWOSDOC.  The  purpose  of  this  test  is  to 
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establish  a  baseline  for  a  student’s  comprehension  level,  as  well  as  to 
indicate  the  effectiveness  of  the  at-sea  curriculum  (Rowden,  2008). 

This  study  evaluates  students’  performance  on  this  test. 

4.  What  measurable  factors  affect  training  effectiveness,  as  measured  by 
student  performance  on  the  pretest  administered  by  SWOS?  The  study 
investigates  the  effect  of  several  factors  (e.g.,  ship  type  and 
commissioning  source)  on  student  performance.  This  information  is 
valuable  as  it  provides  a  broader  picture  of  the  SWOS-at-Sea  training 
beyond  time-to-qualify  and  costs.  SWOS  provided  the  researchers 
with  the  dataset  that  contains  information  for  12  SWOSDOC  classes, 
comprised  of  733  students.  Analysis  of  the  data  includes  descriptive 
and  inferential  statistical  methods.  Descriptive  statistics  present  a 
detailed  analysis  of  the  variables  included  in  the  dataset,  while  the 
multivariate  models  identify  important  determinants  of  test  score 
performance. 

5.  To  what  extent  does  leveling  occur  by  the  end  of  the  3-week 
SWOSDOC  course  of  instruction? 
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II 


Background 


A.  Surface  Warfare  Officer  (SWO)  Qualification  and 
Designation 

OPNAV  Instruction  1412.2H,  updated  in  May  2002,  standardized 
requirements  and  standards  for  the  SWO  qualification  process.  The  instruction 
states  that  all  officers  must  meet  the  following  requirements: 

■  Be  assigned  to  permanent  duty  aboard  a  surface  ship  as  a 
commissioned  officer  for  a  minimum  of  nine  months; 

■  Complete  the  applicable  PQS; 

■  Qualify  and  “serve  successfully”  as: 
o  In-port  Officer  of  the  Deck 

o  Combat  Information  Center  Watch  Officer  or  Surface  Watch 
Officer 

o  Underway  Officer  of  the  Deck 

■  Demonstrate  effective  leadership  skills  and  proficiency  in  performing 
division  officer  duties; 

■  Complete  all  of  the  above  within  the  first  18  months  of  shipboard 
service  unless  granted  an  extension  of  time  authorized  for  up  to  6 
months,  unless  special  circumstances  dictate  otherwise.  (Balisle,  2002, 
p.  3) 

The  instruction  was  amended  in  December  2002,  which  added  that  all  SWO 
candidates  must  complete  the  two-part  SWOSDOC  curriculum,  including  the  SWOS- 
at-Sea  component  and  the  SWOSDOC  3-week  resident  course  (LaFleur,  2002, 
December  29).  The  Surface  Warfare  Directorate  (N76),  Commander  Naval  Surface 
Force,  US  Atlantic  Fleet,  sent  another  message  on  December  16,  2003,  to 
reemphasize  the  importance  for  junior  officers  to  achieve  their  initial  SWO 
qualification:  “Completion  of  Surface  Warfare  Qualification  is  the  critical  component 
of  the  first  division  officer  tour”  (Surface  Warfare  Directorate,  2003,  December  16, 
para.  3). 
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B.  Personnel  Qualification  Standard  (PQS) 

The  PQS  defines  the  minimum  level  of  competency  that  an  individual  must 
demonstrate  to  attain  a  particular  position  or  duty  qualification.  It  is  based  on  a 
collection  of  knowledge  that  demonstrates  skills  required  to  ensure  “safety,  security 
and  proper  operation  of  a  ship,  aircraft  or  support  system”  (Naval  Education  and 
Training  Command,  2004,  p.  7).  The  SWO  PQS  consists  of  three  sections: 
fundamentals,  systems  and  watch  stations.  The  CBT  helps  SWO  candidates 
understand  the  fundamentals  and  ship  systems,  while  their  service  on  watch  stations 
represents  the  application  of  acquired  knowledge.  While  the  PQS  outlines  the 
specific  areas  in  which  a  SWO  candidate  requires  competency,  the  ultimate 
approving  authority  is  the  ship’s  CO;  thus,  a  SWO  candidate  must  demonstrate  a 
proficiency  that  satisfies  the  CO  (Naval  Education  and  Training  Command,  2004). 

C.  Ship  Instruction 

Each  ship  issues  its  own  instruction  to  detail  the  SWO  qualification  process 
for  that  specific  ship.  The  ship  instruction  more  precisely  defines  the  duties  and 
responsibilities  of  both  supervisors  and  SWO  candidates.  For  example,  SWOS 
recommends  that  each  new  junior  officer  be  assigned  both  a  Department  Head  and 
a  technical  mentor.  SWOS  also  recommends  that  a  ship  utilize  the  instruction  to 
provide  specific  accountability  on  the  part  of  the  junior  officer  for  CBT  completion. 
Lastly,  the  instruction  is  designed  to  aid  in  ensuring  a  process  for  consistent 
communication  between  JOs  and  their  supervisors.  However,  the  ship’s  CO 
ultimately  decides  what  particular  instructions  to  promulgate. 

D.  Training  Requirements  Document  ( TRD) 

The  Surface  Warfare  Director  publishes  the  Training  Requirements  Document 
(TRD),  which  defines  the  core  competencies  necessary  to  be  an  officer  at  sea.  This 
document  encompasses  both  Navy  requirements  and  International  Maritime 
requirements  as  specified  in  the  Standards  of  Training,  Certification  and 
Watchkeeping  (STCW)  published  by  the  International  Convention  of  STWC  in  July 
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1995.  This  document  outlines  minimum  qualifications  necessary  to  safely  operate  a 
vessel  at  sea.  The  focus  of  the  TRD  is  navigation,  seamanship  and  shiphandling. 
Specifically,  it  places  the  SWOSDOC  in  the  context  of  the  Navy  SWO  training 
continuum,  which  is  shown  below  in  Figure  1. 
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Areas  (FCA) 

Y 

VR  Shiphandling  Trainer 

Shipboard,  Newport,  Rl  A 
designated  Fleet  Concentration 
Areas  (FCA) 

N 

Certlflcatk>r»s  /  Endorsemerrts 

Shipboard  -  PQS  Based 

N 

Rules  of  the  Road  Examination 

Shipboard  A  Newport,  Rl 

Y 

LEGEND 


NEW  COURSE/  PQS  or  MAJOR  COURSE  REVISION 
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NEW  TRAINER  l  TRAINING  CAPABILITY  REQUIRED 

Figure  1.  SWO  Training  and  Course  Requirements 

(Surface  Warfare  Directorate,  2002,  June  15,  pp.  3-4) 

The  Navy  identified  two  drawbacks  to  the  old  training  regime.  First,  it  was 
inefficient  in  that,  with  a  24-month  qualification  process,  junior  officers  earned  their 
qualification  at  the  end  of  their  initial  sea  tour.  As  a  result,  the  ship  received  very  little 
productive  work  from  the  officer  (in  terms  of  contributions  to  the  ship’s  mission).  It 
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was  estimated  that  officers  completing  the  revised  curriculum  would  receive  their 
qualification  about  12  months  earlier  than  under  the  previous  qualification  process 
(Goodwin,  2003,  February  14,  p.  2-2-2).  A  shorter  time-to-qualify  meant  that  a  junior 
officer  could  begin  to  be  productive  sooner  and  that  the  Navy  could  begin  to  earn  a 
return  on  its  training  investment  earlier. 

An  additional  drawback  to  the  previous  training  approach  was  the  cost.  The 
Navy  incurred  two  moving  costs  for  a  new  Ensign:  the  first  was  the  move  from  his  or 
her  commissioning  source  to  the  SWOSDOC;  the  second  was  the  move  from 
SWOSDOC  to  his  or  her  first  duty  station.  According  to  the  DoN  FY2009  Budget,  the 
Navy  spent  over  $809  million  for  officer  permanent-change-of-station  (PCS)  travel  in 
FY2007  (Office  of  the  Secretary  of  Defense,  2007,  p.  113).  The  cost  per  officer  was 
approximately  $4,616  (based  on  175,270  officer  moves  for  FY  2007).  SWOSDOC 
had  an  annual  throughput  of  902  officers  in  FY  2007.  Under  the  old  training  regime, 
there  would  have  been  a  second  PCS  move  for  902  officers — costing  the  Navy  an 
additional  $4.1  million  annually.^ 

Moreover,  the  cost  of  operating  the  SWOSDOC  schoolhouse  was  a 
consideration.  Gavino  (2002)  indicates  that,  based  on  Naval  Education  and  Training 
Professional  Development  and  Technology  Center  (NETPDTC)  data,  operations  and 
personnel  costs  were  $28,634  per  graduate  in  FY2001  (p.  20).  Thus,  the  cost  of 
operating  the  schoolhouse  under  the  traditional  training  method  was  in  excess  of 
$24  million  for  FY  2001 .  Accounting  for  inflation,  the  cost  of  operating  the 
SWOSDOC  schoolhouse  in  2007  would  have  been  $28.75  million  (using  inflation 
rate  of  15%  (US  Bureau  of  Labor  Statistics,  2008)  under  the  old  training  regime. 

The  actual  cost  of  SWOSDOC  for  the  “leveling  school”  in  FY  2007  was  $3,260  per 
graduate,  or  about  $2.9  million.  While  it  is  beyond  the  scope  of  this  study  to  perform 
a  full  cost-effectiveness  analysis,  these  preliminary  estimates  indicate  that  the  Navy 


^  This  amount  is  based  on  a  calculation  of  an  annual  throughput  of  902  SWOSDOC  students 
multiplied  by  a  per  officer  PCS  move  cost  of  $4,616. 
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is  saving  about  $29  million  annually  in  PCS  and  direct  SWOSDOC  operation  costs 
under  the  new  training  regime.  However,  these  cost  savings  must  be  balanced 
against  the  true  learning  curve  of  Junior  Officers  (JOs),  their  on-the-job  productivity, 
the  cost  of  the  CBT  and  the  true  cost  of  shipboard  on-the-job  training. 

1.  The  New  Curriculum 

By  design,  the  CBT  equips  the  student  with  a  strong  foundation  in  shipboard 
operations.  It  consists  of  six  compact  discs  (CDs)  comprised  of  20  modules  under 
four  main  sections:  division  officer  fundamentals;  navigation,  seamanship  and  ship 
handling;  combat  systems/maritime  warfare;  and  engineering.  Each  section  begins 
with  an  overview  and  related  PQS  items.  The  student  reads  the  appropriate  material 
within  each  module,  and  then  must  accomplish  a  practicum  and  practical  problem 
that  applies  systems  theory  to  the  individual’s  ship.  The  student  completes  a  module 
test  and  receives  a  percentage  score;  80%  is  required  to  pass.  If  the  student  does 
not  pass,  further  review  is  required,  followed  by  a  retake  of  the  exam.  Once  he  or 
she  has  passed  the  module  test,  the  trainee  learns  to  apply  the  learned  principles  in 
a  “real  world”  context  with  a  case  study.  The  process  is  repeated  for  each  module. 
Figure  2  describes  the  training  sequence  of  the  SWOS-at-Sea  training. 

It  is  important  to  note  that  the  SWOS-at-Sea  training  was  designed  to  be 
completed  by  new  Ensigns  working  outside  of  their  other  duties  onboard  their  ships, 
but  within  a  structured  training  time  during  the  work  day.  The  design  also  relied  on 
mentorship  as  a  key  element  of  the  success  of  the  program,  with  goal  setting  for 
program  progression  and  regular  meetings  to  assess  progress  and  make 
adjustments  to  the  Individual  Training  Program.  This  aspect,  along  with  an 
assumption  that  the  CBT  would  be  paired  with  unstructured/unplanned  but  adequate 
on-the-job  training,  represented  a  reasonable  design  for  the  training.  By  definition, 
learning  in  one’s  place  of  work,  with  access  to  the  equipment  and  systems  that  will 
actually  be  used  on  the  job,  should  lead  to  good  transfer  of  learning  between  training 
and  job  performance.  However,  the  present  research  shows  that  the  design  was  not 
fully  implemented  as  intended — which,  in  turn,  led  to  inadequate  preparation  of  the 
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Ensigns  for  their  duties  as  Division  Officers,  frustration  for  the  Ensigns  and  others  in 
their  chain  of  command,  and  less  knowledge  than  intended  when  they  reported  to 
SWOSDOC.  These  findings  are  discussed  further  in  Chapters  IV  and  V. 


One  other  element  of  the  program  that  impacts  the  results  found  in  this  study 
is  the  quality  of  the  CBT.  A  review  of  the  modules  shows  the  following: 

■  The  descriptive  sections  are  very  dry  with  no  video  or  opportunity  for 
user  interaction  to  keep  students  engaged.  They  are  very  much  like 
reading  a  handbook;  definitions  are  provided,  visuals  are  static  and 
include  organizational  charts,  pictures  of  equipment,  equipment  labels, 
etc. 

■  Practical  problems  may  include  case  studies,  which  offer  the 
opportunity  to  see  learning  applications.  However,  the  real  value  of 
case  studies  is  their  use  in  a  collaborative  setting  where  students  can 
work  together  (and  ideally  with  instructor  collaboration)  to  jointly  solve 
problems  and  create  new  knowledge.  The  extent  to  which  individual 
ships  created  opportunities  for  collaborative  learning  is  not  known  but 
is  discussed  further  in  Chapter  IV. 

■  Practicums  direct  students  to  perform  some  activity  or  make  some 
observation.  The  extent  to  which  there  is  this  correspondence 
between  the  CBT  and  OJT  is  not  known  (but  is  also  discussed  in 
Chapter  IV),  and  forms  of  interactivity  that  are  necessary  to  keep 
students  engaged — other  than  multiple  choice  questions  at  the  end  of 
modules — are  not  part  of  the  program. 

As  will  be  shown  in  Chapter  III,  a  large  body  of  literature  in  the  area  of  CBT 
demonstrates  that  these  characteristics  of  the  CBT  are  predictive  of  less-than- 
optimal  learning. 
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Training  Sequence  of  the 
At-Sea  Curriculum 


BracticitmsL 
Practical  Prab: 


6. 


8. 


Initial  Assessment  (recommended) 

—  To  establish  knowledge  base  and  assist  in 
development  of  IndiWdual  Training  Plan 

Develop  Individual  Training  Plan  (ITP) 

CHT  Module  (required) 

—  Cover  theon--  sx-stems  of  100  200  level 
PQS  line  items 

Practicum  (required) 

—  IdentifS*  componeias,  find  references, 
view  procedures 

Practical  Problems  (required) 

—  ‘^Scenario*’  t\pe  questions 

Module  Assessments  (required) 

—  Test  knowledge  learned  in  CBT 

—  809'o  passing  score  required  for 
completion 

Case  Studies  (required) 

—  Applies  knowledge  across  all  modules 

Comprehensive  Exam  (recommended) 

—  "Final"  Exam  to  evaluate  effectiveness  of 
training 


Figure  2.  Training  Sequence  of  the  SWOS-at-Sea  Training 

(Division  Officer  Training  Department,  2005a,  May  12) 

Upon  completion  of  all  modules  and  satisfactory  completion  of  the  applicable 
PQS,  the  CO  grants  the  junior  officer  qualification  as  OOD  U/W.  Unlike  the  previous 
curriculum,  the  junior  officer  is  not  granted  full  SWO  qualification;  the  JO  must  still 
complete  the  3-week  resident  course  at  SWOSDOC  before  achieving  full 
qualification. 

The  Navy  based  the  3-week  SWOSDOC  course  on  the  ship’s  wardroom 
concept.  It  brings  together  a  cadre  of  officers  to  interact  and  learn  from  one  another. 
SWOS  groups  students  into  wardrooms  of  10  to  12  officers  based  on  strengths  and 
weaknesses  to  “level  the  field.”  The  curriculum  is  based  on  six  modules:  damage 
control,  force  protection,  maritime  warfare,  leadership,  navigation,  seamanship  and 
engineering.  The  material  is  presented  in  a  variety  of  formats,  including  classroom 
presentations,  “hands-on”  enactments,  scenario  training  and  simulators. 

SWOS  administers  four  formal  exams  to  students.  The  initial  exam  is  a  four- 
part  pretest  derived  from  the  four  main  sections  of  the  SWOS-at-sea  CBT  that 
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assesses  students’  knowledge.  The  second  test  encompasses  Coast  Guard 
navigational  rules — referred  to  as  the  “rules  of  the  road.”  These  rules  cover  inland 
and  international  navigational  requirements  as  enforced  by  law.  The  third  test  the 
SWOS  administers  covers  tactical  thought  in  areas  of  maritime  warfare.  Lastly, 
SWOS  instructors  observe  and  rate  each  student’s  ship  handling  ability  in  the 
Conning  Officer  Virtual  Environment  (COVE)  simulator.  The  SWOS  records  these 
scores  and  sends  a  copy  to  the  CO  of  a  student’s  ship. 

After  completing  SWOSDOC,  the  student  returns  to  his  or  her  ship,  where 
training  concludes  with  an  appearance  before  a  SWO  qualification  board.  Usually, 
the  board  convenes  within  one  to  two  months  after  the  officer’s  return  to  the  ship. 
Like  the  previous  training  program,  senior,  qualified  SWOs — as  well  as  the  ship’s 
CO — comprise  the  board.  The  board  reviews  the  student’s  qualifications  and  tests 
his  or  her  knowledge.  As  in  the  previous  SWO  qualification  process,  if  the  board 
finds  the  junior  officer  satisfactory,  the  CO  grants  qualification  as  a  SWO,  and  the 
officer  dons  the  coveted  SWO  “pin.” 


Two  years  into  the  new  training  regime,  concerns  began  to  surface  that  the 
self-directed  training  may  not  be  working  as  expected  when  the  DIVOs  were  formally 
tested  at  the  beginning  of  a  3-week  SWOSDOC  “leveling”  school  in  Newport.  It  was 
discovered  that  less  than  3/4  of  reporting  DIVOs  scored  a  passing  grade  on  the  self- 
directed  tests  (i.e.,  based  on  at  least  75%  correct  on  selected  questions  from  the 
DIVO  Fundamentals  Modular  Tests).  Since  that  time,  all  scores  on  these  tests  are 
reported  back  to  the  CO  of  the  junior  officer’s  ship  to  raise  accountability  for  these 
self-directed  training  modules. 


The  purpose  of  this  research  is  to  analyze  how  well  the  reengineered  SWOS- 
at-Sea  training  is  preparing  junior  officers  for  Surface  Warfare  Officer  (SWO) 
Qualification  and  their  jobs  as  division  officers.  The  research  examines  how  well 
junior  officer  SWO  candidates  are  learning  from  the  CBT  and  OJT.  We  adopted 
several  methods  for  the  assessment,  including  interviews,  focus  groups,  and 
statistical  analyses  of  test  scores  of  junior  officers  at  SWOSDOC. 
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Ill 


Literature  Review 


A  large  body  of  research  literature  exists  for  both  computer-based  training 
(CBT)  and  on-the-job  training  (OJT),  which  are  the  key  components  of  the  SWOS-at- 
Sea  training  program.  A  review  of  some  of  the  most  relevant  studies  is  presented 
here  as  a  foundation  for  the  results  of  the  analysis  of  the  present  research. 


A.  Kirkpatrick’s  Evaiuation  Modei 

The  standard  technique  used  to  evaluate  training  programs  has  been  based 
on  Kirkpatrick’s  classic,  four-level  evaluation  taxonomy  (Kirkpatrick,  1998).  The 
taxonomy  proposes  four  levels,  or  steps,  in  evaluating  a  training  initiative:  (1) 
reaction;  (2)  learning;  (3)  behavior;  and  (4)  business  results  (job  applications). 
Phillips  (1997)  added  a  fifth  level  of  evaluation:  return  on  training  investment.  These 
five  levels  involve  the  following  elements  and  evaluation  instruments: 

1 .  Reaction — How  do  the  participants  feel  about  the  training  program? 
Did  they  like  it?  This  element  is  often  measured  via  surveys. 

2.  Learning — What  did  the  training  participants  learn?  Did  they  acquire 
new  knowledge,  skills,  or  attitudes?  This  element  is  often  measured 
via  tests. 

3.  Behavior — Did  the  participants  change  their  behavior  on-the-job?  Did 
they  transfer  the  learning  to  the  job? 

4.  Results — Are  there  any  impacts  on  the  individual’s  or  organization’s 
performance?  For  example,  have  sales  or  productivity  improved? 

5.  Return  on  Investment — Do  the  financial  benefits  of  the  improved 
productivity  (or  sales,  for  example)  exceed  the  costs  of  the  training 
program? 

This  scheme  will  be  used  below  to  assess  the  literature  on  CBT  and  on-the- 
job  training  programs. 
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B.  Computer-based  Training 

A  review  of  the  literature  shows  that  studies  of  CBT  are  found  under  the 
category  of  Distance  Education  (DE),  which  has  evolved  significantly  in  the  past 
twenty  years.  The  literature  differentiates  between  synchronous  learning  (generally 
defined  as  two-way,  simultaneous,  “live”  communication)  and  asynchronous 
learning,  with  which  there  is  no  direct,  “live”  communication  between  the  student  and 
instructor.  DE  can  include  any  learning  with  separation  of  learner  and  instructor 
(place  and/or  time)  during  learning  events,  which  may  include  a  range  of  media  from 
correspondence  courses  to  highly  interactive  web-based  applications.  In  fact,  CBT  is 
now  an  older  technology  as  the  most  current  generation  of  DE  includes  more 
interactive  courses  due  to  technological  advances  and  more  blended  approaches 
that  capture  the  advantages  of  both  mediated  instruction  and  traditional  face-to-face 
instruction. 

Computer-based  training  and  other  forms  of  mediated  instruction  have  been 
used  by  the  military  and  other  organizations  for  many  years.  The  Navy,  for  example, 
received  multi-year  funding  from  the  Defense  Advanced  Research  Projects  Agency 
to  evaluate  the  use  of  CBT  for  Navy  training  (Hurlock  &  Slough,  1976)  over  30  years 
ago.  Then  and  now,  the  military  and  others  continue  to  seek  opportunities  to  convert 
traditional  classroom  education  to  mediated  instruction  in  order  to  save  costs.  Many 
of  these  applications,  if  not  most,  use  mediated  instruction  away  from  schools — 
either  at  centralized  locations  or  on  the  job.  These  applications  allow  the 
organization  to  avoid  travel  costs  (Corporate  Leadership  Council,  2004;  2005)  or 
even,  in  the  case  of  some  military  applications,  to  avoid  permanent  change  of  station 
costs  as  with  the  SWOS-at-Sea  Training. 

Another  frequently-cited  advantage  of  CBT  is  standardization  of  training 
materials,  although  one  study  from  the  Army  Research  Institute  finds  that  when  a 
population  is  diverse  and  people  may  lack  common  knowledge  (as  in  the  military), 
training  needs  to  be  more  tailored.  Giving  the  same  training  to  all  students  is  not  the 
most  effective  approach  (Dyer,  Singh,  Harnam  &  Clark,  2005). 
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Yet  another  advantage  of  CBT  mentioned  in  the  literature  is  the  ability  to  learn 
around  the  clock.  However,  Mackay  &  Stockport  (2006)  point  out  that  high 
operational  tempo  makes  finding  time  to  train  and  educate  difficult. 

Research  on  Navy  CBT  has  historically  focused  primarily  on  enlisted 
personnel  and  A-School  applications.  The  Center  for  Naval  Analysis  (CNA)  recently 
evaluated  CBT  modules  for  Electronics  Technicians,  Fire  Controlmen,  and  Yeomen 
and  found  reductions  in  training  time,  no  impact  on  later  success  at  C-School,  and 
that  self-paced  instruction  can  increase  student  failures.  In  this  configuration,  the 
CBT  was  conducted  in  school  with  instructors  available  to  answer  questions  and 
monitor  progress.  CNA  concluded  that  CBT  may  not  be  appropriate  for  high-risk-of- 
mission-failure  jobs  (Carey,  Reese,  Lopez,  Shuford  &  Wills,  2007). 

In  spite  of  potential  cost  savings  and  many  successful  applications,  the 
results  of  learning  from  CBT  have  been  mixed.  Over  the  years,  many  researchers 
have  questioned  the  balance  of  costs  saved  versus  the  concern  for  a  decrement  in 
the  quality  of  the  instruction  as  compared  to  traditional  face-to-face  learning. 

The  literature  generally  reflects  measures  of  knowledge  outcomes  based  on 
test  scores  and  measures  of  student  satisfaction  from  surveys  taken  by  students 
immediately  after  completing  the  course.  However,  little  is  revealed  in  the  literature 
concerning  changes  in  actual  work  behavior/job  performance  or  about  subsequent 
student  evaluation  of  the  actual  training  results  after  a  longer  period  of  time  has 
passed  (Strother,  2002;  Hui,  Hu,  Clark,  Tam  &  Milton,  2007),  likely  because  these 
things  are  much  more  difficult  to  measure. 

A  study  of  Marine  Corps  training  shows  a  typical  finding:  DE  students  may  do 
as  well  on  grades  and  test  scores  as  their  traditional  classroom  counterparts;  some 
learners  may  like  DE,  but  prefer  face-to-face  contact;  and  transfer  of  student 
knowledge  into  behavior  and  impact  on  the  organization  is  not  always  supported 
(Blevins,  Jones  &  Monroe,  2003).  However,  the  results  are  mixed,  as  noted,  and  this 
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inconsistency  of  data  is  often  due  to  a  large  number  of  factors  that  can  vary  in  CBT 
applications  (Strother,  2002). 

A  study  of  a  DE  application  of  Professional  Military  Education  that  sums  up 
some  of  the  critical  issues  was  conducted  by  the  Government  Accountability  Office 
(GAO,  2004).  The  study  concludes  that  the  DoD  has  not  developed  a  way  to 
properly  evaluate  the  effectiveness  of  the  conversion  of  school-house  materials  to 
DE  applications;  the  Department’s  belief  that  its  results  have  been  positive  are 
anecdotal  versus  metrics-based;  it  has  not  used  systematic  criteria  to  decide  which 
courses  to  convert  to  DE — this  is  done  subjectively;  processes  for  conversion  of 
materials  are  non-standard;  and,  finally,  that  these  problems  are  seen  in  the  private 
sector  as  well. 

Factors  such  as  those  described  above  confound  conclusions  that  can  be 
drawn  about  the  effectiveness  of  DE  and  make  it  difficult  to  predict  the  success  of  a 
given  CBT  application.  However,  those  themes  that  show  consistency  across  many 
studies  are  described  here  to  provide  a  context  to  help  explain  the  results  of  this 
analysis  of  the  SWOS-at-Sea  training  program. 

1.  Student  Satisfaction 

Studies  of  satisfaction  with  CBT  typically  address  two  things:  whether  or  not 
students  like  it  in  comparison  to  traditional  classroom  instruction,  and  their 
impressions  of  the  quality  of  CBT. 

In  a  review  of  e-learning  in  the  corporate  sector,  Strother  (2002)  finds  mixed 
results,  but  concludes  that  a  majority  of  e-learners  preferred  classroom  training  over 
e-learning.  Anderson  (2003)  sites  data  from  a  meta-analysis  that  also  shows  that 
distance  learners  like  DE  but  prefer  face-to-face  contact. 

O’Malley  (1999),  in  his  study  of  undergraduate  business  students,  found  that 
they  did  not  perceive  that  DL  is  as  effective  as  traditional  learning  and  did  not  want 
to  take  more  DL  courses.  They  also  did  not  believe  they  learn  more  in  online 
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courses  and  have  concerns  about  being  able  to  contribute  to  class  discussions. 

They  perceive  online  advantages  to  be  time  savings,  scheduling  flexibility,  and  ability 
to  take  more  courses.  Students  generally  prefer  traditional  courses  to  online 
courses. 

Similarly,  a  GAO  survey  in  2004  found  that  nearly  half  of  distance  learning 
students  in  a  Professional  Military  Education  course  believe  they  are  not  as  well 
prepared  as  the  resident  student  counterparts;  they  were  are  concerned  about  the 
quality  of  their  courses  as  compared  to  those  taken  in  residence. 

Many  reasons  are  recorded  in  the  literature  as  to  why  students  are  less 
satisfied  with  CBT  than  with  traditional  classroom  instruction,  but  they  seem  to 
center  on  less  engagement  with  CBT  (often  due  to  less  interactivity  and 
collaboration,  which  are  discussed  below)  and,  therefore,  less  motivation.  Holden 
and  Westfall  (2006) — in  their  guide  for  medial  selection  for  distance  learning — 
discuss  the  strengths  and  weaknesses  of  each  form  of  instructional  delivery  media. 
For  example,  weaknesses  of  CBT  include  students’  inability  to  interact  with  the 
instructor  by  asking  questions  and,  “Research  has  shown  that  reading  large 
amounts  of  text  on  a  computer  screen  results  in  a  reduction  of  comprehension  and 
speed  when  compared  to  print.  Often  used  as  a  self-study  medium,  students  may 
feel  isolated  and  unmotivated  to  complete  training”  (2006,  p.  23) 

One  factor  shown  to  moderate  this  general  lack  of  satisfaction  with  CBT  as 
compared  to  classroom  instruction  is  self-selection  for  CBT.  Yatrakis  and  Simon 
(2002)  showed  that  students  who  chose  to  take  an  online  course  were  more 
satisfied,  and  they  perceived  that  they  retained  more  information  than  did  the 
students  who  were  not  given  the  choice.  However,  there  was  little  difference  in  the 
grades  of  the  two  groups.  This  implies  that  student  choice  increases  student 
satisfaction  and  perception  of  information  retention,  but  seems  to  make  no  difference 
on  actual  learning  outcomes  (based  on  grades).  What  is  not  addressed  here  is  the 
list  of  other  consequences  of  low  satisfaction  such  as  dissatisfaction  with  the 
organization  sponsoring  the  CBT  and  subsequent  attrition.  Bernard  et  al.  (2004),  for 
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example,  in  their  analysis  of  232  studies,  found  a  higher  dropout  rate  among 
students  who  worked  in  asynchronous  CBT  (individually)  as  compared  to  students 
working  synchronously  and  linked  to  a  classroom  via  videoteleconfrencing.  While 
their  studies  lacked  attitude  data  from  the  attrites,  the  authors  concluded  that  the 
lower  retention  rate  might  be  a  result  of  the  group  affiliation  and  social  pressure  of 
the  classroom-linked  environment.  The  effects  of  collaboration  are  discussed  later  in 
this  chapter. 

There  are  other  factors,  too,  that  may  affect  satisfaction  with  CBT.  Results  of 
a  meta-analysis  by  Allen  et  al.  (2004)  suggest  that  learning  style  plays  a  role  here, 
too.  These  authors  note  that  some  students  may  learn  better  in  a  social  mode,  but 
other  students  learn  better  in  a  solitary  setting;  in  addition,  changing  from  face-to- 
face  learning  to  computer-mediated  communication  can  affect  the  group  outcomes. 

A  related  characteristic  of  some  learners  is  introversion,  which  Abell  (2002),  in  her 
study  of  Army  DE,  suggests  is  more  compatible  with  CBT  than  extroversion.  This 
researcher  also  finds  that  soldiers  perform  better  with  CBT  when  they  know  why 
learning  is  necessary,  can  direct  their  own  learning,  and  can  apply  what  they  have 
learned  to  real-world  problems  (Abell,  2002). 

One  individual  characteristic  that  has  been  discussed  formally  and  informally 
by  many  in  Navy  leadership  is  the  generational  variable.  Because  Generation  Y — 
those  born  between  approximately  1982  and  1994 — grew  up  with  computers,  many 
assume  that  students  from  this  generation  will  be  very  comfortable  with  e-learning. 
Further,  Abell  (2000),  states  that  students  from  both  Generation  X  (born  between 
1965  and  1981)  and  Generation  Y  are  inclined  to  learn  independently  but  adds  that 
they  like  fast-paced,  frequent  feedback  and  visually  intensive  interaction.  Interactivity 
in  CBT  is  discussed  next. 

2.  Interactivity 

Interactivity  in  learning — any  kind  of  learning,  synchronous  or 
asynchronous — is  perhaps  the  most  frequently  mentioned  factor  in  research  on 
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effectiveness  of  DE.  CBT  applications  range  from  rigid  text  combined  with  drill  and 
practice  to  highly  interactive  programs  with  tailored  feedback,  video  and  audio 
components,  ability  to  e-mail  the  instructor  with  questions,  etc.  The  latter  are,  of 
course  most  costly.  Ultimately,  the  highest  level  of  interactivity  can  only  be  found  in 
“blended  solutions,”  in  which  CBT  is  accompanied  by  the  presence  of  an  instructor 
or  mentor  and  built-in  opportunities  for  teamwork,  collaboration  among  peers  and 
instructors,  problem-solving  discussions,  and  hypothesis  testing. 

From  an  instructional  point  of  view  in  CBT,  there  is  no  possibility  for  adapting 
teaching  strategies  to  students  who  are  having  difficulties.  From  the  student’s  point 
of  view,  non-  or  minimally-interactive  CBT  may  simply  be  passive  and  boring 
(MacKay  &  Stockpart,  2006). 

The  work  of  Bernard  et  al.  (2004) — based  on  a  meta-analysis  of  232  studies 
of  both  synchronous  and  asynchnchronous  DE  using  weighted  multiple  regressions 
to  identify  strength  of  predictors  with  respect  to  achievement,  attitudes,  and 
retention — concluded  that  DE  should  incorporate  problem-based  learning  (learning 
material  that  fosters  student  engagement,  deep  processing  and  understanding)  and 
communication/interactivity  (among  learners  and/or  with  instructors;  either  face-to- 
face  or  through  mediation).  The  need  for  student-teacher,  student-student,  and 
student-content  interaction  in  DE  courses  is  discussed  in  detail  by  Anderson  (2003). 

3.  Not  All  Content  is  Appropriate  for  CBT 

An  issue  discussed  by  critics  for  many  years  with  little  resolution  is  the 
question  of  whether  certain  types  of  learning  are  more  suited  for  different  types  of 
instruction.  For  example,  Gagne’s  classic  model  of  “Principals  of  Instructional 
Design”  (Gagne,  Briggs  &  Wagner,  1992)  categorizes  different  types  of  learning  and 
shows  that  different  types  of  learning  events  are  required  for  each.  For  example,  the 
events/instruction  required  for  learning  a  new  attitude  are  very  different  from  learning 
a  procedural  skill.  Similarly,  Hui  et  al.  (2007)  examined  this  issue  with  respect  to  DE 
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language  learning  and  concluded  that  there  were  types  of  learning  better  suited  for 
DE  than  for  traditional  classroom  instruction  and  visa  versa. 

This  fact  has  long  been  recognized  by  members  of  the  training  and  education 
community.  Attempts  have  been  made  for  many  years  by  organizations,  including 
the  Navy,  to  develop  media-selection  algorithms.  Two  recent  efforts  were  conducted 
by  Lee  and  Owens  (2008)  and  by  Holden  and  Westfall  (2006).  Such  devices  require 
experts  to  consider  various  aspects  of  instruction  (e.g.,  content,  the  need  for 
interactivity,  students,  cost,  etc.)  and  to  rate  the  importance  of  various  factors  to 
come  up  with  a  recommendation  as  to  which  medium  is  most  appropriate.  It  is 
interesting  to  note  that  Holden  and  Westfall  (2006)  conclude  that  “finding  the  right 
medium-to-objective  match  will  likely  result  in  a  blended  media  approach”  (p.  29). 

While  there  has  been  little  resolution  on  exactly  which  types  of  skills  are  most 
appropriate  for  CBT  and  other  forms  of  DE,  there  is  general  agreement  that  “soft” 
skills  such  as  leadership  training  are  not  appropriately  taught  via  DE  (GAO,  2004; 

Hui  et  al.,  2007;  Anderson,  2003). 

This  aspect  of  the  literature  is  mentioned  to  underscore  the  complexity  of 
moving  traditional  classroom  instruction  to  DE,  and  the  depth  of  analysis — beyond 
cost  factors — required  before  an  organization  implements  CBT.  Anderson  (2003,  p. 
140)  discusses  how  DE  instruction  choices  are  made  without  adequate  knowledge. 
“For  example,  despite  years  of  study,  it  is  still  unclear  which  students  studying  under 
what  types  of  content  under  what  conditions  and  using  which  instructional  design 
benefit  most  from  synchronous  as  opposed  to  asynchronous  interaction.”  He  further 
notes  that,  “The  search  for  single-faceted  solutions  that  generalize  to  the  many 
diverse  contexts  of  distance  education  is  likely  a  quixotic  quest”  (2003,  p.  141). 

4.  Collaboration 

Collaboration,  also  mentioned  in  the  literature  as  a  “learning  community”  is 
certainly  related  to  interactivity  and  also  frequently  mentioned  as  important  for 
effective  DE.  Hui  et  al.  (2007),  for  example,  define  “learning  community  support  as 
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the  extent  to  which  a  learning  environment  creates  an  active,  strongly  bonded 
community  that  encourages  and  facilitates  knowledge  exchanges  among  peers  and 
their  instructors”  (p.  248).  These  researchers  cite  quantitative  research  that  reports 
the  positive  effects  of  collaboration  on  learning  experiences.  (The  research  of  Allen 
et  al.  (2004)  discussed  earlier  implies  that  some  students — introverts — may  not 
require  all  of  the  benefits  of  a  learning  community.  For  example,  they  may  require 
student-instructor  but  not  student-student  collaboration.) 

In  a  survey  of  MBA  students  taking  online  courses,  Peltier,  Schibrowsky  and 
Drago  (2007)  conclude  that  there  are  interdependent  factors  in  online  education. 
Student-to-student  and  student-to-instructor  communication  is  critical  to  creating  an 
effective  learning  environment.  In  addition,  these  researchers  find  that  the  instructor 
should  create  a  learning  community  rather  than  a  course  in  which  each  student 
operates  independently  of  each  other.  And,  in  a  study  of  e-learning  in  higher 
education,  MacDonald  and  Thompson  (2005)  stress  the  importance  in  DE  of 
collaboration,  interactivity,  and  the  concept  of  a  supportive  learning  community. 

Similarly,  the  Marine  Corps  is  concerned  with  maintaining  cohesion  in  a  DE 
environment  and  sees  that  one  way  to  meet  that  challenge  is  to  provide  mentoring. 

It  feels  that  the  shift  from  an  instructor-centered  to  student-centered  learning 
approach  “must  be  balanced  with  a  team  training  approach  that  is  mentored  and 
proctored  by  senior  leaders”  (Anderson,  2003,  p.  651).  This  aspect  of  the  literature 
review  can  be  nicely  summed  up  by  a  quote  from  Bernard  et  al.  (2004,  p.  38):  “DE 
should  not  be  a  solitary  experience  as  it  was  in  the  era  of  correspondence  courses.” 

5.  Blended  Learning 

The  concept  of  blended  learning  has  existed  at  least  as  long  as  two  classic 
meta-analyses  of  CBT  research  (Kulik,  1991;  Kulik,  1994).  Both  analyses  support 
combining  mediated  instruction  with  classroom  instruction.  Kulik  found  that  students 
typically  learn  more  when  classes  are  supplemented  with  computer-based  tutoring. 
Usually  conceived  as  online  tools  with  live  instructors  (e.g.,  Zimmerman,  2001), 
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blended  learning  is  seen  by  a  majority  of  critics  as  superior  to  CBT.  Mackay  and 
Stockport  (2006)  point  out  that  e-learning  and  classroom  instruction  each  have 
advantages  and  disadvantages,  that  no  one  method  can  directly  replace  the  other, 
and  that  blended  learning  may  offer  benefits  beyond  what  the  other  two  methods 
alone  can  provide. 

Sitzmann,  Kraiger,  Steward  and  Wisher  (2006),  in  their  meta-analysis  of  web- 
based  learning  studies  (generally  considered  superior  to  CBT  because  of  the  greater 
capacity  for  interactivity),  concluded  that  blended  learning  outperforms  either  web- 
based  learning  or  classroom  instruction  alone  with  respect  to  learning,  but  trainees 
seem  to  prefer  classroom  instruction  alone  to  blended  learning. 

6.  A  Final  Consideration 

An  issue  not  addressed  in  this  literature  review  should  also  be  considered. 

Not  surprisingly,  no  study  described  the  effects  of  a  distance  learning  course  being 
mandated  in  addition  to  a  student’s  full-time  job/work  hours,  as  compared  to  a 
distance  learning  course  in  which  students  are  dedicated  full-time  to  school  during 
working  hours.  The  time  spent  by  students  in  the  SWOS-at-Sea  program,  while 
concurrently  performing  jobs  for  which  they  had  inadequate  knowledge,  may  carry 
the  most  weight  in  explaining  the  results  of  the  present  study. 

C.  On-the-Job  Training  (OJT) — Estimating  Return  on 
Investment 

While  this  study  is  concerned  with  CBT,  OJT  is  another  major  component  of 
the  SWOSDOC  curriculum  and  is  conducted  concurrently  with  CBT.  Unlike  CBT, 

OJT  is  not  a  new  development  for  the  SWOSDOC.  In  fact,  OJT  is  one  of  the  oldest 
recognized  training  processes.  Thus,  it  is  important  to  examine  the  characteristics  of 
OJT  as  well  as  its  cost,  benefits,  and  challenges. 
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1.  Return  on  Investments  in  OJT 

Gay  (1974)  approached  OJT  from  a  labor  economics  perspective  as  an 
investment  in  human  capital.  He  calculated  the  returns  to  OJT  as  a  function  of  the 
value  of  marginal  product  over  time.  Figure  3  graphically  illustrates  this  relationship, 
where  “P”  represents  the  end  of  training  (i.e.,  the  end  of  the  investment  period).  The 
shaded  area  to  the  left  of  “P”  is  the  undiscounted  cost  of  OJT — consisting  of  the 
opportunity  cost  of  lost  production  plus  direct  training  costs.  The  shaded  area  to  the 
right  of  “P”  corresponds  to  the  undiscounted  returns  to  OJT  ending  at  “Q”  (the  end  of 
employment).  For  the  organization  to  at  least  breakeven  financially  on  its  investment 
in  training,  the  area  of  returns  (undiscounted  in  this  example)  from  time  P  to  Q  must 
equal  or  exceed  the  area  of  costs  from  time  0  to  time  P. 


Figure  3.  Returns  to  OJT 

(Gay,  1974,  p.  9) 


One  of  the  most  challenging  aspects  to  estimating  return  on  investment  (ROI) 
is  estimating  training  costs.  Numerous  studies  have  examined  the  costs  associated 
with  OJT  (Arzigian,  1967;  Gay,  1974;  Manacapilli,  Bailey,  Beighley,  Bennett  & 

Bower,  2007;  Weiher  &  Horowitz,  1971).  While  the  studies  differ  in  their  cost- 
estimation  methodology,  they  all  note  the  importance  of  foregone  productivity  of  both 
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supervisors  and  trainees.  Since  OJT  takes  place  in  the  work  environment,  and  since 
experienced  workers  conduct  the  training,  their  work  productivity  is  reduced  while 
they  are  engaged  in  training.  Arzigian  (1967)  assumes  that  supervisors  spend  5%  of 
their  time  in  OJT,  thus  calculating  foregone  production  as  pay  multiplied  by  5%. 
While  one  can  argue  the  logic  of  this  assumption,  it  does  serve  as  an  example  of  a 
method  for  including  foregone  production  in  OJT  costs. 

a.  Total  Costs  Related  to  OJT 

A  more  recent  study  by  the  RAND  Corporation  augmented  Gay’s  model  to 
include  initial  skill  training  (1ST)  costs  in  addition  to  OJT  costs  based  on  lost 
productivity  (Manacapilli  et  al.,  2007).  The  expanded  model  gives  a  broader  picture 
of  the  total  costs  as  a  function  of  1ST  and  OJT  costs.  The  authors  refer  to  the 
combined  costs  as  total  human  capital  development  (HOD)  costs  (2007). 

The  researchers  employed  a  four-step  methodology.  First,  they  collected 
relevant  training  data.  Second,  they  conducted  interviews  and  surveys  with  technical 
specialty  supervisors  to  develop  learning  curves  for  each  specialty.  Third,  the 
authors  estimated  the  costs  of  manpower,  instruments,  facilities,  and  time.  Last,  the 
researchers  developed  cost-versus-productivity  curves  for  each  of  the  specialties. 

The  researchers  developed  the  productivity  curves  from  answers  on  a 
questionnaire  distributed  to  E-6s  and  above  in  the  Air  Force.  The  answers  provided 
information  of  how  supervisors  perceive  the  skill  level  of  new  workers  when  they 
complete  1ST  and  how  their  productivity  increases  with  OJT.  They  further  assessed 
how  changing  1ST  course  length  would  affect  overall  training  time  and  associated 
productivity.  The  authors  combined  their  data  to  examine  the  impact  of  1ST  course 
length  changes  on  total  FICD  costs.  Specifically,  the  researchers  examined  the 
implications  of  shifting  the  combination  of  1ST  and  OJT  for  specific  Air  Force  enlisted 
occupations. 

In  particular,  Manacapilli  et  al.  were  concerned  with  defining  a  fully  productive 
airman.  They  used  the  survey  questions  to  construct  the  definition  of  “fully 
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productive.”  They  specifically  asked  supervisors  to  define  a  fully  productive  (i.e., 
mission  capable)  airman  in  each  specialty.  This  information  provided  the  baseline  for 
measuring  each  airman’s  productivity.  Supervisors  were  also  asked  to  define  the 
time  it  took  to  reach  full  productivity. 

The  researchers  used  a  composite  pay  function  to  derive  the  costs.  They 
included  basic  pay,  plus  benefits.  The  authors  did  not  consider  PCS-related  costs. 
The  costs  were  divided  into  two  categories:  the  cost  of  productive  activity  and  the 
cost  of  HCD.  OJT  costs  were  derived  from  a  simple  calculation,  “composite  pay 
times  1  minus  the  productivity  level[...]  OJT  cost  is  based  on  the  cost  of  paying  an 
individual’s  salary  for  non-productive  time”  (Manacapilli  et  al.,  2007,  p.  53).  The 
authors  emphasized  the  importance  of  including  all  of  the  costs  of  OJT.  During  OJT, 
both  the  trainer  and  the  trainee  necessarily  forego  productivity.  Trainees  make 
mistakes  as  they  learn,  which  can  result  in  increased  costs  to  fix  those  mistakes 
(e.g.,  additional  training  time,  repairing  of  equipment). 

The  researchers  constructed  the  learning  curves  based  on  the  productivity 
index  for  each  specialty.  In  other  words,  they  measured  the  number  of  journeymen 
who  could  produce  the  same  output  as  a  given  specialty’s  mix  of  journeymen  and 
fully  qualified  airman  (Manacapilli  et  al.,  2007,  p.  58).  The  total  costs  are  divided  by 
the  total  productivity  to  generate  the  cost  of  one  unit  of  productivity.  This  is  used  to 
measure  how  a  change  in  the  length  of  1ST  would  affect  total  HCD  costs  through 
productivity  units. 


The  researchers  concluded  that  shortening  1ST  results  in;  increased  OJT 
costs,  reduced  productivity,  longer  overall  training  time,  a  lower  experience  ratio, 
and  decreased  standardization  (2007,  p.  7).  Seventy-one  percent  of  the  specialties 
tested  would  significantly  increase  productivity  with  a  10-day  increase  in  1ST  time. 
Increased  productivity  results  in  decreased  costs.  For  example,  the  researchers 
estimated  that  an  increase  in  1ST  for  the  vehicle  maintenance  specialty  would 
decrease  the  overall  HCD  cost  by  $12  million  annually  (p.  70).  In  addition,  the 
authors  found  that  OJT  produces  substantial  costs:  “An  important  implication  is  that 
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future  assessments  of  course  length  adjustments  should  explicitly  consider  effects 
on  the  extent  and  cost  of  OJT”  (p.  71).  Figure  4  graphically  illustrates  these 
conclusions. 


•  Reduced  instructors 

•  Reduced  accessions 

Figure  4.  Effect  of  1ST  &  OJT  Costs  on  HCD  Costs 

(Manacapilli  et  al.,  2007,  p.  7) 

The  graph  depicts  a  nominal  HCD  cost  curve  for  a  typical  airman  who  first 
completes  1ST,  then  enters  the  workforce.  In  this  example,  the  worker  is  20% 
productive  at  the  point  of  entry  into  the  workplace.  As  the  airman  participates  in  OJT, 
he  or  she  increases  in  productivity  until  eventually  reaching  100%  productivity  (at 
about  6.5  years  of  service  (YOS).  In  this  case,  the  authors  reduce  the  amount  of 
time  for  1ST,  which  results  in  a  lower  productivity  upon  entering  the  work  force.  The 
result  is  a  longer  required  period  of  OJT  to  compensate  for  the  shorter  1ST  training 
period.  In  the  example,  OJT  time  is  extended  by  approximately  six  months  (to  a  total 
of  7  YOS).  The  longer  training  time  results  in  increased  OJT  costs.  Since  OJT  costs 
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are  a  function  of  total  HCD  costs,  the  total  cost  of  training  increases.  Conversely, 
lengthening  the  amount  of  time  in  1ST  results  in  an  upward  shift  of  the  curve  and  less 
OJT  time  to  reach  100%  productivity;  thus,total  HCD  costs  are  decreased. 

This  study  is  relevant  to  the  SWOS  training  situation.  The  SWOSDOC 
combines  1ST  and  OJT  such  that  they  occur  simultaneously.  The  results  of  this 
combination  are  uncertain.  It  requires  more  research  to  calculate  the  productivity  lost 
during  OJT,  as  well  as  during  CBT  study  time.  Early  reports  indicate  that  since  the 
combination  of  1ST  (i.e..  Division  Officer  At-Sea  Curriculum)  with  OJT,  total  training 
time  has  decreased  (Rowden,  2008;  Surface  Warfare  Enterprise,  2007).  However, 
the  cost  implications  for  OJT  have  not  been  fully  understood  or  analyzed. 

2.  Challenges  of  OJT 

On-the-job  training  (OJT)  is  defined  as  job-specific  training  conducted  at  the 
work  site  by  a  supervisor  or  an  experienced  employee  using  the  equipment,  tools, 
and  processes  of  the  job  (Steinbach,  2004).  It  is  considered  the  most  common  form 
of  training,  certainly  in  part  because  it  offers  good  potential  for  achieving  “transfer  of 
training” — that  is,  applying  knowledge,  skills,  and  attitudes  learned  to  the  job  (Kim, 
Lee  &  Jacobs,  2001).  It  is  perceived  as  low-cost,  high-return  training  (Sisson,  2001). 

The  literature  differentiates  between  structured  and  unstructured  OJT. 
Structured  OJT  is  training  that  has  been  sufficiently  thought-out  and  planned. 
(Jacobs,  2003).  It  is  formal,  not  informal  training.  It  is  planned  and  delivered  in  a 
systematic  manner,  and  outcomes  and  procedures  are  prescribed  fDeJong, 

Thijssen  &  Versloot,  2001). 

Vernon  (1999)  describes  the  key  advantages  of  structured  OJT  as  its  ability  to 
reliably  and  predictably  achieve  training  objectives  and  its  ability  to  help  build  team 
members..  Jacobs  (2005)  also  finds  that  learners  receive  consistent  training  from 
structured  OJT  because  of  the  actions  taken  before  (preparation/planning),  during 
(standard  delivery),  and  after  (ensuring  learning  has  occurred)  training. 
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Unstructured  (unplanned)  OJT  is  the  traditional  and  most  common  form  of 
OJT.  A  problem  with  this  form  of  OJT  is  that  the  trainer  may  focus  on  completing 
the  work  at  hand  over  conducting  the  training.  Work  provides  the  structure  for  the 
training.  Hence,  tasks  might  occur  out  of  logical  sequence,  or  unusual  events  might 
occur  that  impact  training.  Further,  the  instructor  must  rely  on  his/her  job  experience 
to  do  the  training  (Sisson,  2001). 

In  unstructured  OJT,  training  content  can  be  inaccurate,  incomplete,  or  may 
represent  an  accumulation  of  bad  habits  or  unsafe  shortcuts  on  which  workers  have 
come  to  rely  over  time  (Jacobs,  2005).  Training  can  be  inconsistent.  Even 
experienced  workers  may  use  different  and  unequally  effective  methods  each  time 
they  conduct  the  training,  and  trainers  are  typically  selected  for  their  technical 
knowledge,  without  consideration  of  their  effectiveness  as  an  instructor  (Steinbach, 

2004) .  For  example,  is  the  trainer  a  good  communicator?  Does  the  trainer  know  at 
what  point  to  start  the  training?  Does  he  break  the  job  into  appropriate 
processes/tasks?  Does  he  establish  quality  standards?  Does  he  have  poor  work 
habits?  Does  he  select  and  schedule  adequate  time?  Does  he  provide  appropriate 
feedback?  Instruction  may  not  be  a  priority  for  the  trainer  in  the  context  of  his  own 
work,  and  learners  rarely,  if  ever,  achieve  the  same  training  outcome  (Jacobs, 

2005) . 


Sisson  (2001)  concludes  that  traditional,  unstructured  OJT,  overall,  is 
inconsistent,  inefficient  and  ineffective.  He  states  that  industry  has  learned  that  if 
organizations  do  not  make  a  sufficient  investment  to  ensure  their  OJT  is  planned 
and  executed  properly,  it  often  may  take  on  the  form  of  unstructured  OJT,  which 
according  to  Sisson,  may  be  the  single  most  expensive  training  method  available. 
And  Jacobs  (2005)  takes  the  criticism  further  when  he  claims  that  “Studies  have 
shown  that  unstructured  OJT  leads  to  increased  error  rates,  lower  productivity,  and 
decreased  training  efficiency,  compared  to  structured  on-the-job  training”.  He  further 
states  that  “most  uses  of  unstructured  OJT  are  ineffective  in  achieving  the  training 
objectives,  which  inhibits  the  achievement  of  important  organizational  outcomes”. 
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Another  interesting  aspect  of  the  literature  on  OJT — because  of  its  relevance 
to  the  SWOS-at-Sea  training  is  a  recommendation  for  integrating  OJT  with 
developing  a  Community  of  Learners  or  Community  of  Practice  (CoP).  The  research 
stems  from  a  body  of  knowledge  on  “Situated  learning,”  which  deals  with  the 
concept  thaf’physical  and  social  contexts  in  which  an  activity  takes  place  are  an 
integral  part  of  the  activity,  and  that  the  activity  in  turn  is  an  integral  part  of  the 
learning”  (Stein,  2001).  That  is,  social  interaction  between  trainees  and  others  are 
part  of  the  learning  experience.  Stein  adds,  “Acquiring  knowledge  becomes  a 
collaborative  rather  than  an  individual  process  with  trainees  and  a  work  community 
contributing  to  the  dialogue  about  the  meaning  of  work  situations.  Through  dialogue, 
learning  is  created”  (p.  416).  He  implies  that  situated  learning  can  enhance  planned 
CJT.  “Planned  on-the-job  training  may  come  to  mean  planning  opportunities  on  the 
job  for  trainees  in  a  community  to  reflect  on  their  job  performance  and,  through 
discourse  with  others,  construct  specific  and  situational  workplace  knowledge”  (p. 
422). 


Like  any  training  program,  OJT  presents  challenges.  The  biggest  challenges 
are  lack  of  formalization  and  feedback.  If  not  carefully  monitored,  OJT  can  become 
“following  Joe  around”  (Dertouzos,  Lester  &  Sokow,  1989).  This  can  be  a  challenge 
for  SWOs.  The  operational  tempo  aboard  ship  is  high.  There  are  a  number  of 
activities  and  tasks  to  perform,  and  only  so  much  can  be  accomplished  in  a  24-hour 
period.  The  ship’s  CO  places  heavy  demands  on  the  ship’s  leaders  to  sustain 
superior  performance — the  same  leaders  who  are  responsible  for  mentoring  young 
SWOs.  Under  the  pressure  of  these  demands,  OJT  can  quickly  become  a  “just 
follow  me,”  or  “just  do  what  I  do”  exercise.  Plainly,  the  quality  of  OJT  can  degrade  to 
a  mimicking  exercise  instead  of  a  true  learning  experience.  However,  the  ship’s 
senior  leadership  can  combat  this  tendency  with  clear  objectives  and  measures  of 
effectiveness  to  ensure  that  training  goals  are  accomplished. 

Another  challenge  for  OJT  involves  variability,  which  is  evident  in  several 
facets  of  training.  First,  most  of  the  supervisors  tasked  to  provide  the  training  have 


MANPOWER,  PERSONNEL,  TRAINING  &  EDUCATION  RESEARCH 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY  -  29  - 

NAVAL  POSTGRADUATE  SCHOOL 


no  formal  training  in  instruction  or  adult  learning.  They  must  rely  on  their  own 
individual  experience.  Secondly,  certain  supervisors  may  have  very  little  motivation 
to  train  junior  officers.  Inexperience  and  low  motivation  can  result  in  uneven  training 
that  can  cause  great  variance  in  competency  levels  and  performance  (Mathis  & 
Jackson,  2006).  With  OJT,  leadership  also  runs  the  risk  of  assigning  a  sub-par 
performer  or  worker  with  bad  habits  to  instruct  trainees.  In  the  case  of  the 
SWOSDOC,  relatively  inexperienced  second-tour  division  officers  often  train  SWO 
candidates.  Potentially,  this  might  affect  the  depth  of  training.  In  addition,  low 
motivation  of  the  supervisor  may  leave  a  junior  officer  to  learn  on  his  or  her  own. 

One  researcher  (Jacobs,  2003)  provides  this  illustrative  example: 

When  I  first  reported  to  the  ship,  a  second  tour  DIVO  lieutenant  trained  me. 
We  became  friends.  I  relied  upon  her  to  tell  me  what  to  do  since  I  had  very 
little  training.  She  told  me,  “Just  do  this  or  do  not  do  that.  This  is  how  I  do 
this.”  Eventually  I  learned  what  she  wanted  me  to  learn,  but  I  soon  discovered 
that  I  could  learn  just  as  well  on  my  own.  I  am  still  not  sure  if  I  really  learned 
what  I  am  supposed  to  learn.  Anyway,  after  a  while,  I  just  started  figuring 
things  out  on  my  own.  (p.  22) 

It  is  incumbent  on  the  ship’s  leadership  to  emphasize  the  necessity  of  training 
and  to  monitor  the  training  environment. 

D.  Summary  of  Literature  Review 

The  Navy,  like  other  organizations,  has  a  history  of  converting  traditional 
classroom  training  to  various  forms  of  mediated  instruction,  including  CBT,  to  reduce 
and  provide  learning  at  any  time  and  any  place.  Yet,  many  researchers  have 
questioned  whether  using  CBT  decreases  the  quality  of  instruction.  This  question  is 
difficult  to  answer  because  of  the  many  variables  that  can  impact  outcomes  and  the 
difficulty  in  collecting  measures  of  actual  transfer  of  training  to  job  performance.  In 
fact,  a  GAO  report  criticizes  the  DoD  for  not  developing  adequate  measures  of 
effectiveness  of  distance  education  applications.  Nonetheless,  the  literature  review 
consistently  reflects  several  themes  that  are  relevant  to  the  SWOS-at  Sea  training. 
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Students  are  less  satisfied  and  feel  less  prepared  by  CBT  than  face- 
to-face  learning  (except  for  those  who  self-select  to  take  CBT). 

Interactivity,  collaboration,  and  a  supportive  learning  community  in 
CBT  are  related  to  higher  satisfaction  and  achievement.  Students 
who  must  read  large  amounts  of  text  on  the  screen  are  not  motivated 
to  complete  training. 

o  Successful  CBT,  in  the  Navy  and  elsewhere,  pairs  CBT  with  the 
availability  of  an  instructor  to  answer  questions. 

o  A  higher  level  of  interactivity  is  found  in  “blended  solutions,”  with 
which  the  CBT  is  accompanied  by  an  instructor  or  mentor, 
opportunities  for  collaboration  among  peers  and  instructors, 
problem-solving  discussions,  and  hypothesis  testing. 

o  Generation  Y  students  generally  like  visually  intensive 

interaction  and  fast-paced  instruction  with  frequent  feedback. 

o  Attrition  is  higher  for  students  in  asynchronous  (individual,  not 
real-time)  learning  as  compared  to  synchronous  learning  that 
combines  CBT  with  access  to  instructors  through 
videoteleconferencing. 

Not  all  instruction  is  equal  and,  therefore,  it  is  not  all  equally  applicable 
to  conversion  to  CBT.  The  literature  reflects  a  lack  of  systematic 
analysis  and  criteria  for  deciding  which  courses  should  be  converted 
to  CBT. 

Although  standardization  is  an  advantage  of  CBT,  when  a  population 
is  diverse,  such  as  in  the  military,  the  training  must  be  tailored. 

An  advantage  of  CBT  mentioned  in  the  literature  is  the  ability  to  learn 
around  the  clock.  However,  when  learning  on  the  job,  high  operational 
tempo  makes  finding  time  to  train  and  educate  difficult. 

Students  perform  better  with  CBT  when  they  know  why  learning  is 
necessary,  can  direct  their  own  learning,  and  apply  what  they  have 
learned  to  real-world  problems. 

No  studies  were  found  describing  the  effects  of  a  distance  learning 
course  that  is  mandated  in  addition  to  a  student’s  full-time  job — as 
opposed  to  during  work  hours  or  in  a  full-time  learning  mode — as  is 
the  case  with  the  SWOS-at  Sea  training. 


MANPOWER,  PERSONNEL,  TRAINING  &  EDUCATION  RESEARCH 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


-31  - 


The  Navy  and  most  other  organizations  depend  on  OJT  for  training  to  acquire 
or  maintain  skills,  to  decrease  costs,  and  because  it  presents  the  opportunity  for 
good  transfer  of  training  to  job  performance.  The  literature  is  quite  clear  on  what 
constitutes  effective  OJT: 

■  Structured  (planned  and  systematic)  OJT  is  preferred  to  unstructured 
OJT. 

■  Planned  OJT  produces  consistent  training  that  ensures  training 
objectives  are  achieved. 

■  Unstructured  OJT,  the  more  common  form,  may  be  conducted  by 
someone  who  is  technically  competent  but  not  a  good  instructor. 

■  Opportunities  for  collaboration  are  an  important  aspect  of  OJT 
effectiveness. 

These  finding  are  related  to  the  results  of  interviews  conducted  for  this  study 
described  in  Chapter  IV  and  to  the  analysis  of  the  quantitative  data  described  in 
Chapter  V. 
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IV. 


Interviews  with  SWOs 


To  assess  the  perceived  effectiveness  of  the  SWOS-at-Sea  training, 
Interviews  were  conducted  first  to  collect  background  information  from  four  SWOs  at 
NPS,  six  post-command  SWOs  (all  Captains),  and  with  officers  on  staff  at  SWOS. 
Subsequently,  interviews  were  conducted  with  students  attending  SWOSDOC  in 
May  2008.  The  goal  of  the  interviews  was  to  question  as  many  students  as  possible 
in  the  time  allotted  about  their  experiences  with  the  SWOSDOC  training.  A  total  of 
17  students  were  interviewed  (either  one  at  a  time  or  in  groups  of  2-3)  for  60  to  90 
minutes.  Results  were  not  only  consistent  across  all  officers  interviewed,  but  also 
with  several  articles  that  have  recently  appeared  in  Navy  publications  (Davis,  2008; 
Ewing,  2008;  Shovlin,  2008). 


A.  Results  of  Interviews  with  Captains 

First,  it  is  worth  summarizing  the  themes  from  the  comments  made  by  the 
Captains  who  had  been  commanding  officers  to  show  the  similarity  of  their 
comments  to  those  made  by  the  Ensigns  at  SWOSDOC.  Themes  from  their 
interviews  were: 

■  There  is  inconsistency  across  Commanding  Officers  with  respect  to 
how  much  time  they  put  into  supporting  SWOS-at-Sea  training.  For 
example,  some  hold  a  “SWO-U”;  others  don’t. 

■  Second-tour  DIVOs  don’t  know  as  much  as  they  should  about  the  day- 
to-day  skills  of  running  a  division.  They  are  weak  in  leadership  skills. 

■  The  SWOS-at-Sea  program  sends  a  bad  message  about  the  value  of 
junior  officers  in  the  Surface  Community  as  compared  to  other 
unrestricted  line  officers. 

■  The  CBT  is  not  a  good  product. 

■  In  many  cases,  if  there  is  anyone  helping  the  Ensigns  with  their 
training,  it’s  others  with  not  much  more  experience,  e.g.,  second-tour 
DIVOs. 
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B.  Results  of  Interviews  with  Students  at  SWOSDOC 


The  student  demographics  represented  were: 

■  All  of  the  students  except  two  Limited  Duty  Officers  (LDO)  and  one 
Lieutenant  Junior  Grade  were  Ensigns  (ENS). 

■  Two  of  the  ENS  had  prior  enlisted  experience. 

■  Accession  sources  represented  were  Naval  Academy  (6),  Naval 
Reserve  Officer  Training  Corps  (7),  Officer  Candidate  School  (1),  and 
Seaman  to  Admiral  Program  (1). 

■  The  students’  ship  types  represented  were  DDG  (12),  LPH  (1 ),  CV  (1 ), 
CG  (1),  CGN  (1),  and  FFG  (1). 

■  All  of  the  students  except  two  were  men. 

Content  analysis  was  performed  on  the  interview  data.  The  themes  from  the 
analysis  are  described  in  the  context  of  the  questions  asked. 

The  results  presented  below  will  include  statistical  representation  of  the 
student  responses.  They  will  then  quote  actual  responses  from  the  survey.  By 
highlighting  these  aspects  of  the  data,  the  researchers  hope  to  describe  officers’ 
attitudes  toward  the  CBT  training  approach. 

How  did  you  feel  about  the  SWOS-at-Sea  computer-based  training  (CBT)? 

Fourteen  of  the  17  students  interviewed  did  not  like  the  CBT;  three  said  that  it 
was  “OK.”  Complaints  mentioned  and  comments  made  most  often  converged  on 
the  following: 


Not  engaging,  interactive,  or  interesting 

o  CDs  are  a  cop  out  for  better  training. 

o  The  instructors  here  [at  SWOSDOC]  are  very  involved  and 

engaging.  A  computer  can’t  give  you  that... it  can’t  know  when 
you’re  bored  and  make  it  better.  You  can’t  ask  it  questions. 

o  Am  I  going  to  do  death  by  Power  Point?  No,  I’m  going  to  do 
some  work  that  matters. 
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o 


The  CBT  does  not  make  use  of  available  technology  that  would 
make  it  more  interesting. 

Can  easily  be  gamed  (cheated) 

o  It  was  a  worthless  waste  of  time.  I  cheated  to  get  through. 

o  Have  “they”  gone  through  the  CDs?  Our  XO  looked  at  some 
and  said,  “What’s  this  teaching  you?”  It’s  so  easy  just  to  click 
through  to  get  the  answers. 

Lack  of  time  to  complete  because  of  ship  duties 

o  I  didn’t  make  it  through  the  CBT;  there’s  no  time. 

o  If  you  make  time  to  go  do  the  CBT,  you  are  viewed  as  hiding  in 
your  stateroom. 

o  CBT  just  adds  to  the  burden  of  all  the  hours  required  of  us. 

There  are  accessibility  problems 

o  There  are  lots  of  problems  with  shared  computers. 

They  don’t  address  leading  enlisted  personnel 

o  The  CDs  don’t  prepare  us  to  lead.  I  learned  more  by  being  a 
resident  assistant  in  college. 

They  are  easier  for  some  than  others  because  of  accession  source. 

o  The  lack  of  preparation  for  being  a  Division  Officer  isn’t  as  bad 
(so  the  CBT  isn’t  as  frustrating)  for  Naval  Academy  guys.  In 
NROTC,  we  don’t  get  anything.  I  didn’t  even  know  standard 
commands  when  I  came  onboard. 

Inconsistency  in  accountability  for  completing  the  CBT 

o  It’s  not  standard.  On  some  ships  you  are  held  accountable  for 
completing  the  CBT;  on  other’s  you  aren’t. 


How  do  you  feel  about  learning  from  a  Computer? 

All  17  students  said  they  did  not  like  learning  from  a  computer.  Three  quotes 
sum  up  their  feelings: 
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■  People  still  want  human  contact.  We  may  love  computers  for 
enjoyment,  but  not  for  learning.  There’s  a  reason  why  people  still  go  to 
college  where  there  are  professors  as  opposed  to  Phoenix  on  Line. 

■  I  grew  up  with  computers  as  a  resource  and  for  entertainment  but  not 
to  learn  from.  I  was  a  computer  science  major  and  still  hated  it. 

■  We  need  a  hybrid  approach  where  a  human  is  involved. 

This  finding  was  fascinating  in  light  of  the  generally  held  assumption  that  this 
generation  is  comfortable  with  anything  that  involves  interacting  with  computers. 

Did  you  get  support  from  your  ship  to  compiete  the  CBT? 

This  line  of  questioning  addresses  the  OJT  that  was  designed  to  be  part  of 
the  SWOS-at-Sea  training.  All  students  cited  the  CO’s  attitude  toward  the  training  as 
the  primary  factor  that  determined  whether  or  not  they  got  support  for  the  training. 
Several  of  the  students  had  seen  a  change  in  support  for  the  training  with  a  turnover 
of  COs,  training  officers,  or  Department  Heads.  Regarding  any  help  they  personally 
obtained,  six  students  of  the  total  17  were  on  a  ship  where  the  CO  held  a  “SWO-U.” 
The  SWO-U  may  have  been  implemented  differently  on  different  ships,  but  in  all 
cases,  the  students  found  it  to  be  very  helpful.  Some  comments  follow: 

■  Once  a  week  we  had  SWO-U.  It’s  all  about  good  COs. 

Other  students  were  more  negative  or  at  least  neutral  in  their  responses 
about  the  support  they  got.  The  following  comments  are  examples  of  inconsistency 
in  the  implementation  of  the  OJT: 

■  The  TrainO  is  not  involved.  TrainOs  are  too  busy  with  course 
management  of  all  the  new  systems.  [In  one  case,  a  student  said  he 
was  on  a  ship  that  had  no  Training  Officer  for  1.5  years.] 

■  The  Department  Head  was  sort  of  supportive  in  that  he  said,  ’’Get  it 
done.” 

■  There  are  people  onboard  who  will  help  when  you  ask,  but  not  about 
the  CBT. 
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others  commented  on  the  lack  of  collaboration  among  students  and  between 
students  and  senior  officers; 

■  Everyone  was  just  too  busy,  and  we  were  just  sort  of  left  to  figure 
things  out  by  ourselves. 

In  a  related  question,  15  students  were  asked  if  they  had  a  mentor  onboard 
the  ship.  Twelve  did  not  have  mentors;  three  did.  Of  those  who  did  not,  the  follow 
comment  was  typical: 

■  When  I  went  onboard  I  was  given  a  mentor.  He  showed  me  around 
the  ship  and  that  was  it.  He  was  too  busy  with  INSURVE,  ATG,  etc. 

■  I  was  asked  if  I  wanted  a  mentor.  I  said  yes,  but  they  never  got  back  to 
me.  I  asked  the  DCA  if  he  would  be  my  mentor,  and  he  really  helped. 
Maybe  there  is  a  policy;  I  don’t  know. 

What,  if  anything,  would  you  change  in  the  training  to  make  it  more  effective? 

Thirteen  of  14  students  who  were  asked  this  question  said  that  newly 
commissioned  officers  should  attend  SWOSDOC  before  reporting  to  their  first  ship 
assignments.  Their  reasons  centered  primarily  on  their  frustrations  of  not  feeling 
prepared  to  function  as  Division  Officers.  One  student  said: 

■  We  start  not  knowing  anything;  the  CO  yells  at  us,  and  we  look  dumb 
in  front  of  our  guys. 

■  I  went  into  a  division  with  no  chief,  and  I  needed  1-2  months  on  the  job 
[to  feel  prepared  to  lead  a  division]  but  did  not  get  that. 

■  It  would  be  good  not  to  feel  clueless  and  useless  on  day-one  on  your 
ship. 

■  Help  the  fleet  by  getting  rid  of  the  CBT  and  having  a  school 
upfront... the  trick  is  to  make  it  good  enough  to  where  people  want  to 
go  and  learn  because  they’ve  heard  about  it  from  their  buddies. ..it 
wouldn’t  just  be  a  one-month  party  in  Newport  like  some  people  seem 
to  think  the  old  one  was. 

Eight  of  the  14  said  that  the  CBT  could  serve  as  a  good  reference  if  separate 
from  some  other  primary  training  vehicle  such  as  SWOSDOC. 
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It  was  clear  to  the  researchers  that  there  had  been  talk  among  the  cohort  on 
the  two  above  suggestions,  which  may  have  influenced  the  answers. 

How  long  do  you  intend  to  stay  in  the  Navy  past  your  Minimum  Service 
Requirement  (MSR)? 

While  not  necessarily  related  to  the  SWOS-at  Sea  program,  the  researchers 
were  curious  as  to  the  answer  to  this  question.  Three  said  they  would  stay  to  some 
point  past  the  MSR;  six  said  they  would  definitely  not  stay;  the  two  LDOs  said  they 
would  stay  until  20  years,  and  six  did  not  know  how  long  they  would  stay. 

One  of  the  SWOSDOC  staff  instructors  said  that  he  had  observed  that  the 
Naval  Academy  officers  were  particularly  negative — that  they  are  burnt  out  when 
they  leave  The  Academy  and  almost  hate  the  Navy.  Few  want  to  be  COs.  This 
sentiment  has  been  heard  from  others  with  whom  the  researchers  discussed  this 
topic. 


■  The  CBT  sends  me  the  message  that  the  Navy  doesn’t  care  about  our 
development,  so  I  will  probably  get  out  as  soon  as  I  can. 

C.  Summary  and  Discussion  of  the  Interview  Resuits 

Based  on  the  background  information  collected  from  SWOSDOC  before  the 
interviews  commenced,  the  findings  from  the  interviews  are  not  surprising.  Further, 
based  on  the  literature  reviewed  in  Chapter  III,  the  findings  offer  few  surprises.  In 
other  words,  a  large  body  of  research  findings  reveals  several  elements  of  CBT  and 
OJT  that  are  critical  for  student  satisfaction  and  transfer  of  learning.  If  these 
elements  are  missing,  as  in  the  case  of  the  SWOS-at-Sea  training,  then  negative 
outcomes  for  student  satisfaction  and  achievement  are  predictable.  And  in  fact, 
negative  feelings  about  the  training  are  evident  in  the  interview  results  described 
above,  and  poor  transfer  of  learning  to  tests  administered  at  SWOSDOC  are 
described  in  Chapter  V. 
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A  summary  of  the  interview  results  follows: 


The  majority  of  the  Ensigns  did  not  like  the  CBT. 

The  literature  indicates  that  interactivity  is  positively  related  to 
satisfaction,  and  that  this  may  be  more  pronounced  for  members  of 
Generation  Y. 

The  research  also  shows  that  where  populations  are  diverse,  such  as 
in  the  military,  CBT  needs  to  be  tailored.  For  the  SWOS-at-Sea 
program.  Ensigns  came  to  the  training  with  different  levels  of 
experience  based  on  their  accession  sources,  and  this  added  to  the 
general  level  of  dissatisfaction  for  many. 

Accountability  for  completion  of  the  CBT  was  inconsistent  across  ships 
and  senior  officers,  and  the  system  could  be  easily  “gamed.”  These 
factors,  too,  played  a  role  in  students’  negative  perceptions  of  the 
training. 

Students  did  not  like  learning  from  a  computer. 

The  literature  shows  that,  in  most  cases,  students  prefer  face-to-face 
learning  over  CBT;  this  was  the  case  with  the  SWOS-at-Sea  CBT.  A 
hybrid  learning  solution  that  blends  CBT  with  face-to-face  learning  is 
preferred  over  CBT,  but  this  sort  of  program  was  not  available  to  the 
SWOs.  There  was  not  even  a  capability  for  e-mailing  questions  to  an 
instructor.  This  shortcoming  is  related  to  the  next  finding. 

SWOS-at-Sea  training  was  not  well  supported. 

The  OJT  component  of  the  training  appears  to  have  fallen  short  in 
implementation  as  compared  to  the  original  design  of  the  program.  The 
OJT  can  be  classified  as  unstructured,  which  the  research  literature 
shows  is  less  likely  to  produce  consistent  training  and  to  achieve 
learning  objectives. 

Collaboration,  also,  has  been  shown  to  be  an  important  component  of 
OJT.  The  SWOS-at-Sea  training  provided  some  opportunities  for 
support  and  collaboration  through  the  efforts  of  some  ship  COs  who 
conducted  a  SWOU  and  who  pushed  their  concerns  for  junior  officer 
development  down  the  chain  of  command.  For  the  most  part, 
however,  there  were  few  opportunities  for  students  to  interact  to  share 
knowledge,  experience,  and  insight.  Collaborative  opportunities,  which 
can  help  build  commitment  and  satisfaction,  were  a  function  of  the 
efforts  of  individual  Commanding  Officers  rather  than  a 
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planned/structured  part  of  the  training.  Additionally,  few  students  had 
mentors. 

Time  was  also  an  element  that  detracted  from  consistent  OJT  and 
good  support  of  learning.  The  Ensigns  had  Division  Officer  duties  that 
cut  into  the  time  available  for  the  CBT  and  OJT,  and  many  senior 
officers  were  too  busy  to  help.  To  summarize,  we  have  heard  many 
times  during  the  course  of  this  project  that  “the  SWOS-at-Sea  training 
puts  too  much  of  a  burden  on  the  ship.” 

Ensigns  would  like  training  before  going  onboard  their  first  ships. 

Ensigns  feel  frustrated,  and  in  many  cases  embarrassed,  to  be  put  in 
the  position  of  running  a  division  without  the  requisite  knowledge  and 
skills  to  do  the  job.  They  feel  (and  some  senior  officers  agree)  that  the 
SWOS-at-Sea  training  sends  a  message  to  them  they  are  not  valued 
the  same  as  other  unrestricted  line  officers.  There  was  considerable 
negativity  surrounding  this  topic  with  some  of  the  Ensigns;  they 
indicated  this  was  just  “one  more  reason”  not  to  stay  in  the  Navy. 

Not  all  of  the  SWOS-at-Sea  content  may  be  suitable  for  CBT. 

The  results  described  in  Chapter  V  show  poor  transfer  of  learning  from 
the  CBT  to  tests  administered  at  SWOSDOC.  Many  of  the  possible 
reasons  for  this  have  been  discussed  here,  but  one  other  may  be  the 
suitability  of  the  content  for  CBT.  The  literature  shows  that  not  all 
skills,  knowledge,  and  abilities  can  be  trained  through  mediated 
instruction.  Further,  decisions  to  convert  classroom  instruction  to  CBT 
are  often  made  subjectively  as  opposed  to  analytically. 

The  extent  of  analysis  done  before  converting  the  SWC  training  to 
CBT  is  not  known,  but  this  suitability  factor  should  certainly  be 
addressed  in  any  future  conversions  conducted  by  the  Navy. 
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V.  Statistical  Analysis  of  Student  Performance  on 
Test  Scores  and  Passing  Rates  on  DIVO  Tests 


The  Surface  Warfare  Officers  School  provided  data  for  the  following  statistical 
analyses.  Data  were  also  collected  from  the  Officer  Personnel  Information  Systems 
(OPINS)  to  check  for  consistency  and  to  obtain  additional  demographic  information 
on  each  officer.  The  data  set  consists  of  individual-level  performance  information  for 
junior  officers  participating  in  the  SWOSDOC  training  program  who  completed  the  3- 
week  course  at  SWOS  between  June  2007  and  March  2008.  The  total  number  of 
records  included  in  the  data  set  is  733  and  includes  SWOSDOC  classes  #213 
through  #224. 

We  model  the  determinants  of  the  CBT  test  scores  as  well  as  later  exams 
administered  at  SWOSDOC.  Multivariate  regression  models  are  specified  (based  on 
the  literature)  and  estimated.  Separate  models  are  estimated  for  all  four  CBT 
modules  and  for  the  overall  weighted  CBT  score.  The  determinants  of  test  score 
performance  included  the  following  general  factors;  education,  ability,  motivation, 
demographics,  ship-related  factors,  military  rank,  officer  qualifications,  and 
commissioning  source. 

A.  Computer-based  Training  Test  Scores 

SWOS  administers  a  test  to  students  upon  their  arrival  for  the  3-week  training 
course.  The  test  consists  of  four  modules;  maritime  warfare,  navigation  and 
seamanship,  division  officer  fundamentals,  and  engineering.  A  student  receives  a 
score  for  each  module,  as  well  as  a  cumulative  average  score  for  all  four  modules. 
The  minimum  score  is  zero,  and  the  maximum  score  is  100.  The  test  is  derived  from 
the  CBT  SWOS-at-Sea  training  and  measures  the  effectiveness  of  the  CBT.  Kulik 
and  Kulik  (1991)  emphasized  in  their  meta-analysis  of  computer-assisted  instruction 
that  the  instructional  outcome  most  often  measured  was  student  learning — 
determined  by  performance  on  achievement  examinations. 
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The  metric  adopted  here  for  student  learning  is  student  performance  on  the 
CBT  modules  examination.  Thus,  in  the  analysis  below,  CBT  test  scores  are  used  as 
the  dependent  variable.  Table  1  describes  each  variable  for  each  test  module  and 
for  the  overall  CBT  test.  The  dependent  variable  adopted  is  the  pass  rate  on  each 
individual  exam. 


Table  1.  Computer-based  Training  Dependent  Variables 


Variable  Name 

Description 

Variable  Type 

Cbt 

Average  score  on  the  CBT  four  modules 

continuous 

cbt_mw 

Maritime  Warfare  CBT  score 

continuous 

cbt_nss 

Navigation,  seamanship,  &  ship  handling  CBT  score 

continuous 

cbt_dof 

Division  Officer  Fundamentals  CBT  score 

continuous 

cbt_eng 

Engineering  CBT  score 

continuous 

B.  Demographics 

The  study  combines  the  two  categories  under  the  heading  “Ethnicity,”  due  to 
a  small  number  of  observations  for  minority  (non-Caucasian)  groups,  which  is  shown 
in  Figure  5.  Therefore,  this  research  uses  either  race  or  ethnicity  to  identify  members 
of  a  minority  (non-Caucasian)  group.  Furthermore,  this  study  recodes  ethnicity  to  a 
binary  variable  due  to  its  distribution.  Members  are  classified  as  Caucasian  or 
“Other.” 
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Distribution  of  Variable-  Ethnic 
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Figure  5.  Race/Ethnic  Distribution 


Gender  is  another  important  demographic  factor  that  may  affect  academic 
performance  (Schram,  1996).  Schram’s  (1996)  meta-analysis  of  18  quantitative 
performance  studies  found  that  women  slightly  outperformed  men  in  post-secondary 
psychology,  education,  and  business  courses. 

C.  Military  Factors 

Military  factors  capture  the  officer’s  military  service  prior  to  arriving  at  SWOS. 
The  data  include  commissioning  source,  rank,  prior  enlisted  service  and  SWOSDOC 
class.  Since  CBT  takes  place  on  each  officer’s  ship,  and  the  environment  on  each 
ship  type  differs,  it  is  likely  that  performance  on  the  CBT  will  differ  across  ship  type. 
The  selected  variables  will  reflect  the  learning  environment  and  work-related 
conditions  that  will  affect  the  officer’s  ability  to  study  onboard  the  ship.  These 
variables  include  time-onboard  (in  months),  ship  name,  ship  type  and  homeport. 
Other  important  variables  that  we  use  capture  an  officer’s  shipboard  qualifications. 
These  variables  indicate  whether  the  officer  qualified  as  EOOW  and  the  time  in 
months  that  an  officer  has  been  qualified  OOD  U/W.  Time-onboard  describes  the 
total  time  that  an  officer  has  been  assigned  to  his  or  her  ship.  It  is  measured  in 
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months,  begins  from  report  date  and  ends  with  the  month  the  officer  reports  to 
SWOSDOC.  Since  CBT  is  self-paced,  it  is  assumed  that  less  time-onboard  equates 
to  a  higher  motivation  level.  Thus,  time-onboard  is  used  as  a  proxy  for  motivation. 

The  ship  name,  ship  type  and  homeport  characterize  the  specific  ship  to 
which  an  officer  is  assigned.  The  original  data  included  observations  for  13  ship 
types.  The  study  combines  the  ships  according  to  major  category  of  ship.  For 
example,  the  two  different  types  of  amphibious  assault  ships  (LHD  &  LHA)  are 
combined  into  the  category,  “amphibious.”  However,  the  study  combines  the 
auxiliary  ship  category  with  the  mining  ship  category  (i.e.,  ships  whose  mission  is 
mining  and  mine  sweeping)  due  to  a  low  number  of  observations  (less  than  3%  of 
the  sample).  The  two  graphs  in  Figure  6  show  the  distribution  of  officers  by  ship 
type  before  and  after  the  ship  categories  were  combined  into  5  groups. 


NAVAL  POSTGRADUATE  SCHOOL 


MANPOWER,  PERSONNEL,  TRAINING  &  EDUCATION  RESEARCH 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 


-44- 


Distribution  of  Ship  Type 
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Figure  6.  Ship  Type  Distribution  Before  and  After  Variable  Conversion 

The  final  two  ship-related  variables  reflect  the  qualifications  obtained  by  an 
officer  before  attending  SWOSDOC.  Since  each  officer  must  be  OOD-qualified 
before  reporting  to  SWOSDOC,  OOD  is  a  continuous  variable  measured  as  the 
number  of  months  required  to  achieve  the  OOD  qualification.  Lastly,  some  officers 
have  qualified  for  engineering  officer  of  the  watch  (EOOW)  in  addition  to  OOD.  A 
binary  variable  is  created  to  indicate  whether  the  officer  qualified  EOOW.  The  study 
measures  the  impact  of  this  additional  qualification  on  CBT  performance.  Table  2 
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describes  the  variables  of  the  shipboard  environment  and  their  hypothesized  effects 
on  test  scores. 


Table  2.  Modified  Ship  Factors 


Variable  Name 

Description 

Variable 

Type 

Code  for 
Categorical 
Variables 

Hypothesized 
Effect  on  CBT 
Test  Score 

Ship/Qualifications 

tobmonth 

Months  onboard 
ship 

continuous 

+ 

oodmonth 

Months  qualified 
OOD  U/W 

continuous 

ship cat 

The  Category  of 
Ship 

categorical 

auxmine 

Combination  of 
Auxiliary  &  Mining 
ships 

dummy 

Other=0 

auxmine=1 

cruiser 

All  types  of 
Cruisers 

dummy 

Other=0 

cruiser=1 

+ 

destroyer 

All  types  of 
Destroyers 

dummy 

Other=0 

destroyer=1 

+ 

frigate 

All  types  of 
Frigates 

dummy 

Other=0 

frigates=1 

+ 

amphib 

All  types  of 
Amphibious 
Assault  ships 

dummy 

Other=0 

amphib=1 

eoow 

Engineering 
Officer  of  the 
Watch  Qualified 
(Y/N) 

dummy 

No=0 

Yes=1 

+ 

D.  Education  Variables 

Studies  of  student  performance  normally  include  educational  background 
information.  The  correlation  between  previous  academic  performance  and  future 
performance  is  well  documented  (Anderson  &  Benjamin,  1994;  Butcher  &  Muth, 
1985;  Chisholm,  Cobb  &  Kotzan,  1995).  Eskew  and  Faley  (1988)  found  a  statistically 
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significant  effect  of  previous  college  grades  on  a  specific  college  accounting  course. 
Kulik  and  Kulik  (1991)  found  that  all  of  the  254  studies  they  reviewed  included 
previous  academic  achievement  variables  to  control  for  academic  ability  and 
motivation.  The  data  for  this  study  contained  information  on  an  officer’s 
undergraduate  education,  major,  and  college  attended.  However,  the  data  did  not 
include  information  on  undergraduate  grade  point  average  (GPA).  We  obtained  data 
from  OPINS  on  individual  officer  undergraduate  GPA,  but  information  was  available 
for  only  5%  of  the  sample.  As  a  result,  we  created  the  following  variables  to  control 
for  academic  ability:  undergraduate  classification,  type  of  undergraduate  school,  and 
undergraduate  major.  Following  Bowman  and  Mehay  (2002),  we  constructed 
undergraduate  classification  from  Barron’s  Profiles  of  American  Colleges,  which 
classifies  colleges  and  universities  according  to  their  admissions  competiveness. 
Barron’s  (2006)  uses  averages  of  the  factors  of  entrance  criteria  (i.e.,  SAT  scores, 
class  ranking,  and  percentage  of  freshman  admitted).  Each  university  is  ranked 
along  a  competiveness  scale  ranging  from  “Noncompetitive”  to  “Most  Competitive” 
(2006).  For  example.  Harvard  University  is  ranked  “Most  Competitive”  since  it  has 
very  strict  entrance  criteria,  while  Norfolk  State  University  is  ranked  as  “Less 
Competitive”  since  it  accepts  the  majority  of  students  that  apply.  A  plus  (+)  is  added 
to  a  ranking,  with  the  exception  of  the  “Most  Competitive”  ranking.  The  latter 
indicates  schools  that  fall  within  a  certain  category  but  that  also  limit  the  amount  of 
students  accepted.  Barron’s  also  includes  a  “special”  classification  for  colleges  with 
specialized  fields  of  study  (e.g.,  art  schools).  Figure  7  shows  how  the  observations  in 
this  data  set  are  categorized  using  the  Barron’s  classification  system. 

“Special”’  colleges  are  dropped  since  they  comprise  less  than  2%  of  the 
sample  (only  four  observations)  and  they  represent  certain  talents  (e.g.,  artistic 
ability)  along  with  academic  ability,  which  may  confound  their  affects  on  CBT  score. 
Similar  to  Bowman  and  Mehay  (2002),  the  competitive  categories  are  grouped  into 
three  classifications:  (1)  Least  Competitive  (combines  Noncompetitive,  Less 
Competitive,  and  Competitive);  (2)  Competitive  (combines  Competitive  +,  Very 
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Competitive,  and  Very  Competitive+);  and  (3)  Most  Competitive  (combines  Highly 
Competitive,  Highly  Competitive+,  and  Most  Competitive). 


Distribution  of  Barron's  Classifications 


Figure  7.  Barron’s  Competitive  Categories 


In  addition,  we  distinguish  between  the  type  of  undergraduate  school 
attended:  public,  private,  or  USNA.  Bowman  and  Mehay  (2002)  found  that  officers 
from  private  schools  received  better  performance  reviews  from  superiors.  Brewer, 
Eide,  and  Ehrenberg  (1999)  also  concluded  that  there  are  significant  economic 
returns  to  attending  a  private  college.  While  this  information  does  not  reflect 
differences  in  student  academic  performance  between  public  and  private 
universities,  it  may  be  useful  to  see  if  there  are  differences  associated  with  school 
ownership.  For  example,  private  universities  may  have  more  resources  and, 
therefore,  yield  improved  cognitive  skills  of  their  graduates. 

Moreover,  there  is  likely  to  be  a  difference  between  students  educated  at  the 
USNA  and  students  educated  at  other  universities.  Naval  Academy  students  receive 
specialized  training  that  includes  professional  development  coursework  and  OJT 
during  summer  cruises,  and  they  are  also  exposed  to  a  full-time  military  environment 
during  college  (Bowman  &  Mehay,  2002).  In  addition,  students  receive  this 
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education  in  a  Navy-centric  environment.  Thus,  this  study  controls  for  these 
differences.  Figure  8  shows  the  distribution  of  USNA  students  compared  to  those 
educated  at  other  private  or  public  universities. 
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Figure  8.  Distribution  of  College  Type 

In  summary,  Barron’s  classification  and  type  of  university  serve  as  proxies  for 
student  ability  and  cognitive  skills. 

We  also  classify  an  officer’s  undergraduate  major  into  one  of  five  categories: 
natural  science,  social  science,  mathematics  concentration,  arts,  and  business. 
Natural  science  includes  the  physical  sciences  (i.e.,  chemistry,  biology, 
mathematics,  etc.),  while  social  science  encompasses  the  study  of  human 
individuals  and  groups  (i.e.,  economics,  education,  sociology,  etc.).  Mathematics 
concentration  is  based  on  degree  programs  such  as  engineering,  computer  science, 
and  mathematics  that  emphasize  calculations,  measurements,  and  problem  solving. 
Flumanities,  general  liberal  art  degrees,  and  fine  art  degrees  comprise  the  arts 
category.  Lastly,  business  includes  those  majors  with  a  focus  on  business 
administration  or  business-related  subjects  such  as  accounting  or  marketing.  Figure 
9  displays  the  distribution  of  officer’s  undergraduate  education  in  the  data  set. 
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Distribution  of  Officer  Undergraduate  Major 
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Figure  9.  Distribution  of  Undergraduate  Major 

Once  DIVOs  pass  their  first  warfare  qualification,  OOD-U/W  (Officer-of-the- 
Deck-Underway),  and  achieve  a  pass  rate  of  80%  correct  on  each  of  the  four  self- 
directed  tests,  they  are  sent  to  the  3-week  classroom  training  program  at  SWOS, 
called  the  “leveling”  school.  The  goal  of  this  3-week  program  is  to  assure  that  all 
DIVOs  reach  the  same  competency  level  in  the  knowledge  base  contained  in  the 
self-directed  training  modules.  During  the  first  day  of  training,  all  DIVOs  are 
administered  selected  questions  from  the  self-directed  training  package.  The  scores 
on  this  exam  are  weighted  to  arrive  at  a  total  score  of  DIVO  Fundamentals,  with 
Maritime  Warfare  and  Engineering  modules  receiving  the  highest  weights  and  the 
Division  Officer  Fundamentals  module  receiving  the  lowest  weight.®  In  addition  to  the 


®  These  weights  are  based  on  a  linear  regression  model  that  regresses  the  combined  score  on  the 
scores  reported  for  the  four  modules.  The  results  of  this  statistical  estimation  technique  produced  the 
the  following  weights,  which  are  attached  to  each  score:  Maritime  Warfare  =  0.303;  Engineering  = 
0.278;  Navigation-Seamanship  =  0.238;  and  Division  Officer  Fundamentals  =  0.172. 
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weighted  total  score,  SWOSDOC  also  assigns  a  passing  rate  of  75%  correct,  which 
is  somewhat  lower  than  the  80%  passing  rate  that  is  required  during  execution  of  the 
self-directed  tests  (CBT)  taken  aboard  ship.  The  lower  pass  rate  of  75%  allows  for 
some  depreciation  of  knowledge  that  may  have  occurred  since  the  junior  officers 
studied  the  modules  aboard  their  ships  many  months  earlier. 

E.  Descriptive  Statistics 

We  analyzed  data  on  12  classes  that  entered  the  “leveling”  school  between 
July  2007  and  March  2008.  The  data  was  provided  by  SWOSDOC  and  covered 
class  numbers  213  through  224.  The  average  total  test  score  and  the  average 
percent  of  officers  who  achieved  a  “passing”  score  for  the  twelve  classes  are  shown 
below  in  Figure  10. 

Two  trends  are  noticeable  in  Figure  10.  First,  the  average  weighted  total 
score  for  DIVOs  has  risen  over  time — peaking  at  almost  80%  for  Class  218  (see 
Graph  A).  More  important,  however,  is  the  continuous  rising  trend  in  the  percent  of 
students  achieving  a  passing  grade  over  this  period,  which  rose  from  75%  to  84%. 
The  reason  these  two  trends  differ  is  evident  if  we  compare  the  distribution  of  the 
weighted  total  scores  for  early  compared  to  later  classes.  We  first  notice  the  class 
sizes  are  similar  (56  versus  58),  but  the  standard  deviation  for  the  earliest  Class  214 
(7.494)  exceeds  that  for  the  most  recent  Class  222  (6.692)  (See  Graph  B).  Far 
more  individuals  from  Class  214  scored  “below  passing,”  while  more  enrollees  of 
Class  222  scored  at  or  above  passing.  Both  of  these  characteristics  result  in  far 
higher  pass  rates  than  in  higher  average  test  scores  between  the  two  classes.  In 
essence,  SWOSDOC  appears  to  be  directed  at  raising  the  level  of  common 
knowledge  required  for  successful  surface  warfare  qualification — that  is,  in  achieving 
the  “leveling  effect,”  which  is  the  goal  of  the  3-week  classroom  training  program.  All 
analysis  reported  in  this  study  will,  therefore,  focus  on  the  “passing  rates”  on  the 
DIVO  tests  as  a  measure  of  achievement  in  both  self-directed  learning  as  well  as  in 
classroom  learning  environments. 


NAVAL  POSTGRADUATE  SCHOOL 


MANPOWER,  PERSONNEL,  TRAINING  &  EDUCATION  RESEARCH 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 


-  51  - 


While  much  of  the  analysis  in  this  study  focuses  on  the  weighted  total  test 
scores,  secondary  analyses  are  carried  out  on  the  passing  rates  of  the  four  modular 
tests  administered  on  the  first  day  of  SWOSDOC.  These  rates  for  the  classes  are 
shown  below  in  Figure  10.  The  highest  passing  rates  are  on  Navigation-Seamanship 
and  Engineering,  which  both  average  around  80%.  In  contrast,  only  40%  to  60% 
pass  the  Division  Officer  Fundamentals  Module,  and  only  40%  pass  the  Maritime 
Warfare  module.'' 


(A) 

AVERAGE  SCORE  AND  AVERAGE  PASSING  RATE 
ON  DIVISION  OFFICER  FUNDAMENTALS  EXAM 


Figure  10(A).  Total  DIVO  Fundamentals  Exam  Scores  and  Selected 

Class  Distributions 


''  It  may  be  noted  that  SWOSDOC  has  chosen  the  latter  two  modular  scores  to  be  central  to  the  index 
score  used  to  identify  high-achieving  DIVO  test  scores.  They  are  combined  with  the  score  on  the 
Rules-of-the  Road  exam  administered  at  SWOSDOC  to  create  the  index  score.  Clearly,  SWOSDOC 
views  relatively  high  scores  on  Maritime  Warfare  and  Division  Officer  Fundamentals  as  indicators  that 
distinguish  DIVOs  from  their  fellow  classmates.  This  is  not  to  say  that  doing  well  on  Engineering  and 
Navigation-Seamanship  is  not  important.  On  the  contrary,  these  are  extremely  important 
competencies  for  all  DIVOs.  Perhaps  so  much  effort  is  put  forth  toward  these  two  competencies,  that 
officers  can  be  distinguished  from  each  other  most  easily  based  on  the  scores  on  Maritime  Warfare 
and  Division  Officer  Fundamentals. 
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DISTRIBUTION  OF  PASSING  RATES  ON  DIVO  FUNDAMENTALS  EXAM 


CLASS  214 


CLASS  222 


Figure  10(B).  Total  DIVO  Fundamentals  Exam  Scores  and  Selected 

Class  Distributions 


PASSING  RATES  ON  MODULAR  EXAMS  BY  CLASS 


Figure  11.  Passing  Rates  on  DIVO  Modular  Exams:  Classes  213-224 
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VI 


Statistical  Analysis  of  Test  Results 


Data  for  all  732  enrollees  for  the  DIVO  course  between  July  2007  and  March 
2008  were  obtained  from  SWOSDOC  and  were  used  to  analyze  the  self-directed 
learning  achieved  in  the  program,  along  with  the  classroom  training  during  the  3- 
week  leveling  school.  The  analysis  focuses  on  commissioned  officers.  Differences 
between  LDO-CWO  and  commissioned  officers  require  that  the  two  groups  be 
analyzed  separately. 

We  first  describe  the  differences  between  non-commissioned  and 
commissioned  officers  and  then  estimate  the  impact  of  these  differences  on  the 
passing  rates  of  the  various  DIVO  Fundamentals  tests.  We  then  analyze  in  more 
detail  the  impacts  of  Navy  assignments  and  individual  characteristics  on  the  test 
pass  rates.  In  particular,  the  analysis  of  commissioned  officers’  performance  on  self- 
directed  training  focuses  on  four  main  determinants:  (i)  ship  type  and  home  port 
assignments,  (ii)  commissioning  source  and  college  quality,  (iii)  academic  major,  and 
(iv)  demographic  factors.  The  results  of  this  analysis  are  presented  according  to 
these  four  dimensions  to  explore  the  major  factors  that  appear  to  affect  the  ability  to 
learn  information  that  is  crucial  in  the  training  of  junior  surface  warfare  officers. 

For  the  SWOSDOC  classes  convening  between  July  2007  and  March  2008, 
more  than  15%  (118  out  of  a  total  of  732)  of  the  Division  Officers  attending  the  3- 
week  classroom  training  program  were  non-commissioned  officers  (either  Chief 
Warrant  or  Limited  Duty  Officers).  These  DIVOs,  with  few  exceptions,  do  not  have  a 
college  degree.  Table  3  highlights  important  background  differences  between  the 
two  groups. 
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Table  3.  Characteristics  of  SWOSDOC  Classes  213-224 


VARIABLE 

VALUES 

COMMISSIONED 

LDO-CWO 

COUNT 

PERCENT 

COUNT 

PERCENT 

ETHNIC  GROUP: 

1  White 

443 

78.8 

81 

74.3 

2  AfAmer 

37 

6.6 

8 

7.3 

3  Hispanic 

23 

4.1 

6 

5.5 

4  AsianPI 

30 

5.3 

11 

10.1 

5  Other 

29 

5.2 

3 

2.8 

Total 

562 

100.0 

109 

100.0 

Missing 

52 

9 

GENDER: 

Male 

477 

77.7 

115 

97.5 

Female 

137 

22.3 

3 

2.5 

Total 

614 

100.0 

118 

100.0 

GRADE: 

Ensign 

545 

89.1 

55 

56.1 

LT-LCDR 

67 

10.9 

43 

43.9 

Total 

612 

100.0 

98 

100.0 

Missing 

2 

20 

SHIP  TYPE: 

1  DDG 

226 

36.8 

31 

26.3 

2  CG 

94 

15.3 

10 

8.5 

3  FFG 

110 

17.9 

12 

10.2 

4  AMPHIB 

163 

26.5 

24 

20.3 

5  SMCLF 

20 

3.3 

18 

15.3 

6  CVN 

1 

0.2 

23 

19.5 

Total 

614 

100.0 

118 

100.0 

HOME  PORT: 

1  Atlantic 

308 

50.2 

49 

41.5 

2  West  Pac 

226 

36.8 

53 

44.9 

3  Japan 

57 

9.3 

11 

9.3 

4  Other 

23 

3.7 

5 

4.2 

Total 

614 

100.0 

118 

100.0 

YEAR  GROUP: 

<2006 

95 

15.5 

58 

49.2 

2006 

470 

76.5 

44 

37.3 

>2006 

49 

8.0 

14 

11.9 

Total 

614 

100.0 

116 

100.0 

Missing 

2 

PRIOR  SERVICE: 

0  None 

453 

81.5 

0 

0.0 

1  1-7Yrs 

25 

4.5 

5 

5.0 

2  8-12Yrs 

66 

11.9 

29 

28.7 

3  13-16Yrs 

11 

2.0 

46 

45.5 

4  17-24Yrs 

1 

0.2 

21 

20.8 

Total 

556 

100.0 

101 

100.0 

Missing 

58 

17 

TIME  FROM  OOD-U: 

1  1  mo 

292 

50.7 

41 

38.7 

2  2  mo 

147 

25.5 

18 

17.0 

3  3  mo 

74 

12.8 

12 

11.3 

4  4+  mos 

63 

10.9 

35 

33.0 

Total 

576 

100.0 

16 

100.0 

Missing 

38 

12 

TIME  ON  BOARD: 

1  1-11  mos 

114 

19.8 

18 

17.0 

2  12-18  mos 

400 

69.4 

31 

29.2 

3  19+ mos 

62 

10.8 

57 

53.8 

Total 

576 

100.0 

106 

100.0 

Missing 

38 

12 
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PERCENT 


(A) 


AVERAGE  SCORE  ON  DIVISION  OFFICER  FUNDAMENTALS  EXAM: 
LDO/CWO  vs.  COMMISSIONED  OFFICERS 


(B) 


AVERAGE  PASSING  RATE  ON  DIVISION  OFFICER 
FUNDAMENTALS  EXAM:  LDO/CWO  vs.  COMMISSIONED  OFFICERS 


|—  LDO/CWO - COMMISSIONED] 


Figure  12.  DIVO  Fundamentals  Total  Score  and  Passing  Rates  for 
Commissioned  Officers  versus  LDO-CWOs 


PER 
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Table  4  shows  that  LDO/CWOs  differ  from  commissioned  officers  in  the 
following  attributes: 

■  They  are  overwhelmingly  male  (98%,  versus  78%  for  commissioned 
officers); 

■  They  are  more  likely  to  be  LTJG  or  higher  (44%,  versus  1 1  %  for 
commissioned  officers); 

■  The  are  less  likely  to  be  assigned  to  a  CRUDES  ship  (35%,  versus 
52%  for  commissioned  officers),  especially  carriers  (20%,  versus  0% 
for  commissioned  officers); 

■  They  are  from  earlier  year  groups  (49%  before  2006,  versus  16%  for 
commissioned  officers); 

■  They  qualified  OOD-U/W  earlier  (33%  qualified  at  least  4  months  prior 
to  enrolling  in  SWOSDOC,  versus  11%  for  commissioned  officers);  and 

■  They  spent  longer  time  onboard  ship  (54%  spent  over  18  months 
onboard,  versus  only  11%  for  commissioned  officers). 

Figure  12  above  also  shows  that  non-commissioned  officers  have  lower 
weighted  total  test  scores  (Panel  A)  and  lower  passing  rates  on  the  total  self- 
directed  tests  (Panel  B)  compared  with  commissioned  officers.  There  is  roughly  a 
10-point  difference  in  passing  rates  between  the  two  groups,  although  it  is  of  interest 
to  note  the  gap  has  shrunk  in  the  later  classes. 

Because  non-commissioned  officers  appear  to  be  so  different  from 
commissioned  officers,  their  experiences  in  learning  in  self-directed  OJT 
environments  compared  with  classroom  environments  are  analyzed  separately.  To 
examine  differences  between  the  non-commissioned  and  commissioned  officers, 
two  non-linear,  logistic  regression  models  are  specified  and  estimated.  The  first 
model  includes  only  a  binary  variable  for  non-commissioned  officer  status  (LDO- 
CWO)  and  omits  other  important  factors  that  also  may  predict  who  passes  the  DIVO 
Fundamentals  Test  (which  combines  questions  from  all  four  CBT  modules).  The 
marginal  effect  of  each  factor  is  reported  in  the  first  column,  and  the  significance 
level  is  reported  in  the  second  column  of  Table  4. 
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The  impact  of  non-commissioned  status  on  the  likelihood  of  passing  the 
weighted  score  on  all  four  self-directed  DIVO  tests  in  general  mirrors  that  of 
commissioned  officers.  While  it  first  appears  NCO’s  are  nearly  20%  (in  Row  1)  less 
likely  to  pass  the  self-directed  tests  when  the  scores  are  combined  (i.e.,  Total  CBT 
Score),  this  difference  turns  out  to  be  due  mainly  to  their  ship  assignments.  When 
ship  assignments  are  included  in  the  full  model  specification  (Columns  3  and  4) 
those  assigned  to  amphibious  ships  and  carriers  are  21%  and  40%,  respectively, 
less  likely  to  pass.  Over  14  of  all  non-commissioned  officers  are  assigned  to  these 
ships — compared  with  only  30%  of  commissioned  officers.  In  essence,  what  at  first 
appears  to  be  a  difference  due  to  non-commissioned  status  turns  out  to  be  related  to 
the  type  of  ships  to  which  non-commissioned  officers  are  more  likely  to  be 
assigned — especially  carriers,  on  which  passing  rates  on  the  self-directed  tests  are 
abysmally  low.  The  only  DIVO  Fundamental  Module  in  which  non-commissioned 
officers  were  less  likely  to  pass  than  commissioned  officers  is  the  Navigation- 
Seamanship  module  and,  once  again,  ship  type  differences  most  likely  are  the 
cause  of  the  disparity  in  these  passing  rates. 
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Table  4.  Effect  of  CWO/LDO  on  Passing  Rates  on  DIVO 
Fundamental  Tests  and  Exams 


CWO-LDO 

Full-model 

CBT  or  Classroom  Exam 

Specification  Only 

Specification 

Marg.  Eff. 

Sig.  Lev. 

Marg.  Eff. 

Sig.  Lev. 

Total  CBT  Score 

-.198 

.014 

-.126 

.145 

Maritime  Warfare  Module 

-.042 

.390 

-.029 

.730 

DIVO  Fundamental  Module 

-.035 

.484 

+.017 

.848 

Engineering  Module 

-.028 

.522 

-.030 

.570 

Navigation-Seamanship 

Module 

-.158 

.001 

-.098 

.211 

Maritime  Warfare  Exam 

-.115 

.005 

-.128 

.003 

COVE  Exam 

-.047 

.297 

-.008 

.913 

During  the  SWOSDOC  classroom  training,  a  final  exam  is  administered  to  all 
DIVOs  on  the  intricacies  of  maritime  warfare.  Here,  we  see  non-commissioned 
officers  are  roughly  13%  less  likely  to  pass  the  exam  (i.e.,  with  at  least  75%  correct 
answers)  than  commissioned  officers — even  when  we  controlled  for  ship 
assignments  as  well  as  other  factors.  This  difference  supports  the  hypothesis  that 
those  who  previously  shunned  college  do  not  learn  as  well  as  those  who  spent  four 
or  more  years  competing  in  a  college  classroom  environment.  We  also  note  that 
training  for  the  COVE  exam  is  not  truly  done  in  a  traditional  classroom  environment 
but  rather  in  a  one-on-one  simulator  training  environment;  in  this  case,  there  is  no 
apparent  performance  differences  between  non-commissioned  and  commissioned 
officers  noted. 

In  summary,  the  results  above  support  the  concept  of  a  dual  classroom 
training  program  separating  DIVOs  by  non-commissioned  status.  It  is  possible  that 
non-commissioned  officers’  performance  on  classroom  tests  may  improve  when  they 
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do  not  have  to  compete  with  college-educated  officers  in  a  classroom  environment. 

It  is  also  noted  that  their  ability  to  learn  in  self-directed  learning  environments  is 
similar  to  that  of  commissioned  officers,  with  observed  differences  more  related  to 
ship  assignments  than  to  personal  attributes. 

A.  Ship  Type  and  Home  Port  Assignments 

Ship  type  and  home  port  assignment  is  not  a  random  process  for  many 
commissioned  officers.  Graduates  of  the  Naval  Academy  and  NROTC  programs  are 
assigned  a  graduation  rank  order  according  to  their  academic  and  military 
performance  over  their  college  years.  If  graduates  with  higher  rankings  share  similar 
preferences  for  ships  (e.g.,  CRUDES  ships  over  others)  and  home  ports  (e.g.,  San 
Diego  and  Hawaii  over  others),  groups  from  these  ships  and  ports  may  have  better 
grades  on  both  self-directed  CBT  tests  and  classroom  DIVO  exams.  Thus,  ship  type 
and  home  port  assignments  may  be  indirectly  related  to  DIVO  testing  results  due  to 
the  self-selection  process  characteristic  of  Navy-funded  scholarship  students.  Figure 
13  below  shows  the  average  passing  rate  for  all  SWOSDOC  classes  by  ship  type 
(Panel  A)  and  home  port  (Panel  B). 

1.  Ship  Type 

Of  the  551  SWOSDOC  enrollees  who  were  commissioned  officers  and  had 
valid  information  on  personal  characteristics  and  Navy  experience,  roughly  1/3  were 
assigned  to  destroyers  and  1/4  to  amphibious  ships.  Less  than  20%  were  assigned 
to  either  cruisers  or  frigates,  and  less  than  5%  to  small  boats  or  auxiliary  ships. 

Panel  A  of  Figure  13  classifies  these  differing  ship  types  into  three  groups,  including 
cruiser-destroyers,  amphibious  ships  and  small/auxiliary  ships  (“other”).®  Of  the  two 
largest  ship  type  assignments,  we  note  that  it  is  not  uncommon  for  the  passing  rate 


®  There  were  only  three  commissioned  officers  assigned  to  carriers;  thus,  this  ship  type  is  not 
represented  in  the  following  analysis.  It  may  also  be  noted  that  the  statistical  models  are  constructed 
with  those  assigned  to  destroyers  as  belonging  to  the  comparison  group — with  others  assigned  to 
cruisers,  frigates,  amphibious  and  small/auxiliary  ship  types. 
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on  the  weighted  total  CBT  tests  of  those  from  CRUDES  ships  to  be  20%  to  30% 
points  higher  than  those  from  amphibious  ships.  The  small  number  of  officers 
assigned  to  the  third  group  of  ships  (other)  results  in  highly  variable  passing  rates 
over  the  classes,  and,  on  average,  the  passing  rate  is  close  to  that  of  officers  from 
CRUDES  ships. 


(A) 

AVERAGE  PASSING  RATE  OF  COMMISSIONED  OFFICERS  ON 
DIVISION  OFFICER  FUNDAMENTALS  EXAM  BY  SHIP  TYPE 


Figure  13(A).  Passing  Rates  on  Total  DIVO  Fundamentals  Exam 
by  Ship  Type  and  Home  Port  Assignments 
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(B) 

AVERAGE  PASSING  RATE  FOR  COMMISSIONED  OFFICERS 
BY  HOME  PORT  ASSIGNMENT 


Figure  13(B).  Passing  Rates  on  Total  DIVO  Fundamentals  Exam 
by  Ship  Type  and  Home  Port  Assignments 

Multivariate  models  are  next  specified  to  estimate  the  marginal  effect  of  ship 
type.  In  the  first  stage,  the  model  controls  only  for  ship  type  and  includes  binary 
variables  for  four  ship  types:  cruisers,  frigates,  amphibious,  and  small/auxiliary  ships 
Next,  the  model  is  expanded  by  adding  control  variables  for  commissioning  source, 
quality  of  college/university,  academic  majors,  demographic  variables  (ethnicity  and 
gender)  and  various  Navy  experience  variables — including  time  since  OOD-U/W 
qualification  and  SWOSDOC  attendance,  total  time  onboard  ship,  and  years  of  prior 
enlisted  experience.® 

Selected  results  from  this  modeling  process  are  shown  below  in  Table  5.  For 
each  initial  CBT  test  and  SWOSDOC  exam,  we  estimate  the  independent  marginal 


®  Recall  that  these  models  are  estimated  on  samples  that  include  only  commissioned  officers.  Of  the 
551  commissioned  officers  included  in  the  analysis  sample,  roughly  18%  were  prior  enlisted 
personnel. 
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effect  of  the  four  ship  types  (compared  to  the  omitted  category  of  destroyers). 
Statistically  significant  coefficients  are  highlighted  with  bold  letters. 

Table  5.  Effect  of  Ship  Assignment  on  Passing  Rates  on  DIVO 
Fundamentai  Tests  and  Exams 


Marginal  Effects  &  Significance  Leveis  of  Ship  Assignment 


CBT  or 

Cruisers 

Frigates 

Amphibious 

Smaii/Auxiliary 

Classroom  Exam 

Marg. 

Effect 

Sig. 

Lev. 

Marg. 

Effect 

Sig. 

Lev. 

Marg. 

Effect 

Sig. 

Lev. 

Marg. 

Effect 

Sig. 

Lev. 

Total  CBT  Score: 

Ship  Type  Only 

+.001 

.987 

-.074 

.252 

-.248 

.000 

-.264 

.053 

Full  Specification 

-.005 

.939 

-.035 

.561 

-.225 

.000 

-.129 

.309 

Maritime  Warfare 
Moduie: 

Ship  Type  Only 

-.038 

.559 

-.148 

.010 

-.252 

.000 

-.218 

.011 

Full  Specification 

-.031 

.626 

-.111 

.051 

-.230 

.000 

-.128 

.233 

DIVO 

Fundamental 

Module: 

Ship  Type  Only 

-.011 

.867 

-.126 

.057 

-.115 

.040 

-.488 

.001 

Full  Specification 

-.006 

.923 

-.106 

.087 

-.116 

.032 

-.156 

.200 

Engineering 

Moduie: 

Ship  Type  Only 

Full  Specification 

-.020 

.719 

-.016 

.754 

-.026 

.563 

+.098 

.233 

Navigation- 

Seamanship 

Moduie: 

Ship  Type  Only 

-.115 

.086 

-.059 

.305 

-.088 

.077 

-.060 

.587 

Full  Specification 

-.078 

.192 

-.055 

.312 

-.079 

.096 

-.106 

.372 

Maritime  Warfare 
Exam: 

Ship  Type  Only 

Full  Specification 

+.009 

.835 

+.002 

.956 

-.030 

.415 

-.199 

.099 

COVE  Exam: 

Ship  Type  Only 

Full  Specification 

-.073 

.244 

-.093 

.119 

-.071 

.164 

-.021 

.859 
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The  estimated  marginal  effects  of  cruisers  (vs.  destroyers)  indicate  that  the 
only  significant  difference  in  passing  rates  for  self-directed  CBT  test  scores  is  found 
in  the  Navigation-Seamanship  module  (in  which  those  on  frigates  are  11.5%  less 
likely  to  pass  the  initial  test  module).  However,  once  other  control  variables  are 
added  to  the  model  specification,  this  difference  is  no  longer  statistically  significant. 
One  possibility  to  account  for  this  overall  similarity  in  passing  rates  is  that  both  ship 
platforms  incorporate  AEGIS  weapons  systems — making  the  environment  for 
learning  aboard  these  ships  the  same. 

The  passing  rates  of  officers  from  frigates  (having  no  AEGIS  systems)  differ 
only  from  those  assigned  to  destroyers  in  two  self-directed  training  modules — 
Maritime  Warfare  (-14.8%)  and  DIVO  Fundamentals  (-12.6%).  The  magnitude  of 
these  estimated  marginal  effects  is  only  slightly  lower  once  all  other  controls  are 
added,  though  the  differences  remain  statistically  significant. 

The  most  pervasive  difference  in  passing  rates  for  the  self-directed  training 
modules  is  found  for  those  assigned  to  amphibious  ships,  where  differences  are 
statistically  significant  for  three  of  the  four  self-directed  training  modules  (all  but 
Engineering),  as  well  as  in  the  weighted  total  score  for  all  four  modules.  While  the 
difference  in  passing  rates  varies  from  -8%  (Navigation-Seamanship)  to  -23% 
(Maritime  Warfare)  for  the  three  training  modules,  the  overall  difference  in  the  total 
score  of  the  four  modules  is  estimated  to  be  -22.5% — even  after  all  control  variables 
are  included  in  the  model.  These  results  suggest  that  the  main  reason  for 
differences  in  passing  rates  for  officers  aboard  amphibious  ships  is  due  to  the  varied 
learning  environments,  as  opposed  to  differences  in  schooling  ability  or  the  length  of 
time  onboard  ship  or  the  time  between  the  OOD-U/W  qualification  and  attendance  at 
SWOSDOC. 


Differences  in  passing  rates  of  the  overall  weighted  CBT  score,  along  with  the 
Maritime  Warfare  and  DIVO  Fundamentals  modules,  for  officers  assigned  to 
small/auxiliary  ships  (e.g.,  mine  sweepers  and  combat  logistics  ships)  compared  to 
those  assigned  to  destroyers  are  significant  only  in  the  first  model.  The  estimated 
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marginal  effects  for  small/auxiliary  ships  are  reduced  when  control  variables  are 
added,  and  they  become  statistically  insignificant.  These  results  suggest  that  while 
assignment  to  these  ship  types  is  correlated  with  lower  passing  rates  on  self- 
directed  training  modules,  most  of  the  differences  are  due  to  schooling-related  ability 
differences  rather  than  to  differences  in  the  learning  environment  across  differing 
ship  types. 

Two  additional  exams  are  administered  toward  the  end  of  the  3-week 
SWOSDOC  program;  one  in  a  traditional  classroom  environment  (Maritime  Warfare 
Exam)  and  the  other  in  a  simulator  one-on-one  environment  (Conning  Officer  Virtual 
Environment,  or  COVE).  Here,  we  find  no  significant  difference  in  passing  rates  (i.e., 
where  passing  requires  a  minimum  score  of  75%  correct  for  the  Maritime  Warfare 
Exam,  and  a  score  of  4.0  out  of  5.0  on  the  COVE  exam)  by  ship  type.^  Most 
interesting  is  the  finding  that  officers  assigned  to  amphibious  ships  are  23%  less 
likely  to  pass  the  self-directed  Maritime  Warfare  training  module  but  have  nearly 
identical  probabilities  of  passing  the  end-of-SWOSDOC  Maritime  Warfare  exam, 
which  is  given  in  a  traditional  classroom  environment.  One  possible  explanation  for 
these  differences  may  lie  in  the  role  personal  motivation  plays  in  the  self-directed 
training  environment.  DIVOs  aboard  amphibious  ships  most  often  do  not  experience 
the  same  command  hierarchy  with  respect  to  Surface  Warfare  officers,  as  most  of 
their  superiors  are  from  non-surface  warfare  communities.  It  is  likely  that  very  little 
pressure  or  expectations  are  made  on  amphibs  with  regard  to  passing  the  self- 
directed  CBT  training  modules.  To  the  extent  this  is  true.  Navy  ship  assignment 
policies  must  ensure  that  all  DIVOs  receive  at  least  one  assignment  aboard 
CRUDES  ships  during  their  initial  two  DIVO  tours.  Other  assignment  possibilities 
could  include  the  assurance  of  a  “quality  spread”  across  ship  types  for  the  first  ship 
assignment;  this,  however,  would  constrain  the  self-selection  process  based  upon 


^  The  only  exception  to  this  finding  is  the  20%  lower  probability  of  passing  the  COVE  exam  for  those 
assigned  to  small/auxiliary  ships.  One  possibility  is  that  little,  if  any,  prior  experience  on  these  ships  is 
related  to  personal  navigation  compared  to  experiences  of  all  other  ship  types. 
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academic-military  performance  rank  order  for  graduates  of  the  Naval  Academy  and 
OCS. 


2.  Home  Port  Assignment 

As  indicated  earlier,  graduates  of  USNA  and  NROTC  self-select  not  only  their 
ship  type  but  also  their  home  port.  This  is  not  to  imply  that  graduates  from  these 
commissioning  sources  always  get  their  first  pick  for  both  assignments,  but  only  that 
they  may  be  more  likely  to  be  assigned  to  either  their  preferred  ship  type  or  their 
preferred  home  port  location.  The  lower  panel  of  Figure  13  above  categorizes 
respondents’  home  port  assignments  into  three  main  groups:  Atlantic  Coast  (50%  of 
all  assignments).  Western  Pacific  (37%  of  all  assignments,  including  Hawaii),  and 
Japan  (9%  of  all  assignments).  A  final  home  port  assignment  (others)  exists  in  the 
data,  but  its  numbers  are  too  small  to  chart.  Casual  observation  of  the  chart 
indicates  that  officers  assigned  to  the  West  Coast  generally  have  higher  passing 
rates  on  the  overall  score  of  self-directed  training  modules,  while  those  from  the 
Atlantic  Coast  have  relatively  lower  passing  rates. 

To  analyze  the  effect  of  home  port  assignments,  we  estimate  logit  models  of 
self-directed  and  classroom  training  tests  that  include  binary  variables  for  home  port. 
Those  assigned  to  the  Atlantic  Coast  are  treated  as  the  comparison  group.  Binary 
variables  for  the  West  Coast,  Japan,  and  Other  Ports  are  the  only  variables  included 
in  the  first  model.  A  second  model  is  then  specified  that  adds  ship  type, 
commissioning  source,  and  gender.  This  second  specification  seeks  to  determine 
how  the  effect  of  home  port  assignment  on  passing  rates  changes  once  we  control 
for  commissioning  source  and  ship  type.®  Since  gender  preferences  for  ships  and 
home  ports  also  may  be  important,  a  binary  variable  for  gender  is  also  included  in 
this  second  specification.  Finally,  a  third  model  is  specified  that  includes  college 


®  For  the  three  large  home  port  groups  (Atlantic,  Western  Pacific,  and  Japan),  roughly  50%  of  all 
ships  were  CRUDES.  Twice  as  many  amphibious  ships  are  “home  ported”  in  Japan  than  in  the 
Atlantic  or  Western  Pacific  bases,  while  the  Other  Home  Port  (comprised  of  Ingleside,  TX, 
Pascagoula,  MS,  and  Sardina)  had  no  amphibious  ships,  23%  frigates  and  77%  small/auxiliary  ships. 
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quality,  academic  major,  and  various  variables  that  capture  the  junior  officers’ 
experiences  aboard  ships  prior  to  their  arrival  at  SWOSDOC. 

The  results  of  these  three  models  for  each  self-directed  CBT  test  and  the  two 
end-of-SWOSDOC  exams  are  shown  in  Table  6.  The  results  indicate  that  very  little 
of  the  difference  in  passing  rates  on  the  self-directed  training  modules  is  explained 
by  home  port  assignments.  The  major  exception  is  the  finding  that  home  port  affects 
passing  rates  on  the  DIVO  Fundamentals  module.  However,  these  differences  are 
either  not  affected  or  become  exaggerated  once  additional  control  variables  are 
added  to  the  model.  Officers  assigned  to  Western  Pacific  bases  or  to  Japan  are 
+7.8%  and  +12.6%,  respectively,  more  likely  to  pass  this  important  self-directed 
module  than  those  assigned  to  Atlantic  Coast  bases.  Once  ship  type,  commissioning 
source,  and  gender  are  included  in  the  model  specification,  passing  rate  differentials 
for  officers  assigned  to  the  “other  base”  category  jump  from  +3.4%  to  +32.1%  and 
become  highly  significant. 

Identifying  the  reasons  for  this  difference  in  passing  rates  within  a  self- 
directed  training  environment  is  beyond  the  scope  of  this  study.  However,  we  can 
eliminate  the  factors  that  are  included  in  the  model  as  explaining  the  remaining 
differential  in  passing  rates.  That  is,  these  differences  in  this  self-directed  module 
are  not  due  to  differing  ship  types,  to  self-selection  occurring  from  commissioning 
source  and  college  quality,  or  to  gender  or  ethnicity  differences.  Instead,  other 
possible  reasons  include  the  OPTEMPO  scheduling  differences  (“dwell  time”)  that 
may  exist  across  home  ports,  or  the  expectations  and  training  environments  set  by 
each  ship’s  commanding  officer.  Clearly,  further  analysis  is  required  to  understand 
why  this  difference  exists. 
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Table  6.  Effect  of  Home  Port  on  Passing  Rates  on  DIVO 
Fundamental  Tests  and  Exams 


CBT  or 

Classroom  Exam 

Marginal  Effects  &  Significance  Levels  of  Home 
Port  Impacts: 

Western 

Pacific 

Japan 

Other 

Marg. 

Effect 

Sig. 

Level 

Marg. 

Effect 

Sig. 

Level 

Marg. 

Effect 

Sig. 

Level 

Total  CBT  Score: 

Home  Port  Only 

+.036 

.386 

-.016 

.815 

+.030 

.751 

Ship-Source — 
Gender  Added 

+.071 

.088 

+.063 

.315 

+.182 

.111 

Full  Specification 

NS 

NS 

NS 

Maritime  Warfare 
Moduie: 

Home  Port  Only 

+.040 

.380 

+.005 

.949 

-.006 

.956 

Ship-Source — 
Gender  Added 

NS 

NS 

NS 

Full  Specification 

NS 

NS 

NS 

DIVO 

Fundamental 

Module: 

Home  Port  Only 

+.082 

.066 

+.101 

.149 

+.034 

.747 

Ship-Source — 
Gender  Added 

+.081 

.076 

+.127 

.067 

+.309 

.007 

Full  Specification 

+.078 

.099 

+.126 

.080 

+.321 

.002 

Engineering 

Moduie: 

Home  Port  Only 

+.026 

.490 

-.016 

.805 

+.041 

.623 

Ship-Source — 
Gender  Added 

+.037 

.324 

+.008 

.900 

-.173 

.424 

Full  Specification 

NS 

NS 

NS 
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CBT  or 

Classroom  Exam 

Marginal  Effects  &  Significance  Leveis  of  Home 
Port  Impacts: 

Western 

Pacific 

Japan 

Other 

Marg. 

Effect 

Sig. 

Level 

Marg. 

Effect 

Sig. 

Level 

Marg. 

Effect 

Sig. 

Level 

Navigation- 

Seamanship 

Moduie: 

Home  Port  Only 

+.032 

.342 

+.068 

.156 

-.033 

.700 

Ship-Source — 
Gender  Added 

NS 

NS 

NS 

Full  Specification 

NS 

NS 

NS 

Maritime  Warfare 
Exam: 

Home  Port  Only 

+.017 

.559 

-.004 

.929 

-.118 

.209 

Ship-Source — 
Gender  Added 

+.013 

.655 

+.003 

.955 

+.035 

.726 

Full  Specification 

NS 

NS 

NS 

COVE  Exam: 

Home  Port  Only 

+.035 

.358 

+.090 

.110 

+.086 

.275 

Ship-Source — 
Gender  Added 

+.031 

.422 

+.095 

.086 

+.198 

.012 

Full  Specification 

NS 

NS 

NS 

Another  noteworthy  finding  in  Table  6  is  the  lack  of  statistical  difference  in 
passing  rates  for  the  exams  administered  at  the  end  of  the  SWOSDOC  program.  No 
differences  in  passing  rates  for  the  traditional  classroom  training  program  (Maritime 
Warfare)  are  found  in  any  of  the  models,  and  the  positive  differential  in  passing  rates 
of  the  COVE  exam  for  officers  assigned  to  Japan  and  the  Other  Category  become 
insignificant  once  schooling  ability  (i.e.,  college  quality  and  academic  majors)  and 
experience  aboard  ships  are  included  in  the  model  specification.  In  summary,  the 
effect  of  home  port  assignment  on  the  learning  environment  of  self-directed  training 
aboard  ships  is  supported  by  the  data  analysis  but  is  not  related  to  more  traditional 
classroom  learning  found  at  SWOSDOC. 
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B.  Commissioning  Source  and  College  Quality 

As  discussed  earlier,  graduates  of  the  Naval  Academy  and  Naval  ROTC  units 
are  given  a  rank  order  according  to  their  academic  and  military  performance  as 
midshipmen.  Individuals  with  higher  rank  order  get  to  choose  their  ships  and  related 
home  ports  accordingly;  thus,  one  must  control  for  commissioning  programs  when 
analyzing  the  performance  on  self-directed  and  classroom  training  tests  and  exams. 
In  addition,  while  the  study  lacks  data  on  the  academic  ability  of  individuals,  it  is 
possible  to  approximate  general  levels  of  academic  ability  according  to  the  quality  of 
the  school  attended.  In  this  study,  the  college  or  university  of  each  officer  is  linked  to 
the  quality  index  taken  from  Barron’s  Guide  to  Colleges  and  Universities. 

1.  Commissioning  Source 

The  first  set  of  statistics  analyzes  the  difference  in  passing  rates  on  these 
tests  and  exams  between  graduates  of  the  Naval  Academy  and  others  who  attended 
“most  selective”  colleges  either  as  a  NROTC  student  or  an  OCS  graduate.®  The 
reason  this  is  done  is  to  determine  the  unique  impact,  if  any,  on  the  probability  of 
passing  the  DIVO  Fundamental  tests  of  Academy  graduates  as  compared  to 
graduates  of  equally  selective  schools.  In  essence,  these  tests  approximate  the 
difference  in  test  scores  between  an  individual  qualified  for  entry  into  USNA  with 
those  who  are  equally  qualified,  but  who  instead  chose  to  attend  a  similarly  selective 
civilian  college  or  university.  In  this  data  set,  there  are  200  Annapolis  graduates  and 
a  total  of  63  other  commissioned  officers  who  attended  “most  selective”  colleges  (56 
NROTC  graduates  and  7  OCS  graduates).  The  passing  rates  on  the  weighted  total 
self-directed  CBT  scores  for  Academy  versus  other  graduates  of  “most  selective” 
schools  are  plotted  in  Figure  14. 


®  As  explained  in  more  detail  in  the  following  section,  there  are  roughly  24  colleges/universities 
included  in  this  category  of  colleges  in  the  Barron’s  Guide  to  Colleges  and  Universities.  Schools  with 
the  largest  number  of  officers  include:  Notre  Dame  (11),  Cornell  University  (6),  University  of  Southern 
California  (5),  and  the  University  of  Pennsylvania  (4).  Other  schools  in  this  category  are  MIT, 
Carnegie  Mellon,  Vanderbilt,  and  the  University  of  Virginia,  among  others. 
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AVERAGE  PASSING  RATE  FOR  COMMISSIONED  OFFICERS 
FROM  "MOST  SELECTIVE"  COLLEGES 
BY  COMMISSIONING  SOURCE 


Figure  14.  Average  Pass  Rate  By  College  Selectivity 


While  the  average  passing  rate  of  Academy  graduates  rises  from  60%  to  over 
80%,  we  notice  many  classes  of  NROTC  and  OCS  graduates  of  most  selective 
colleges  have  higher  passing  rates  than  USNA  graduates.  We  now  turn  to  statistical 
analysis  of  individual  performance  on  self-directed  and  traditional  classroom  tests 
and  exams  for  this  sub-set  of  DIVOs.  The  results  of  the  two-stage  modeling  of 
passing  rates  are  presented  in  Table  7. 
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Table  7.  Effect  of  Commissioning  Source  on  Passing  Rates  on  DIVO 
Fundamental  Tests  and  Exams 


CBT  or  Classroom 
Exam 

Marginal  Effects  &  Significant  Levels 
of  Commissioning  Source: 

NROTC* 

ocs* 

Marginai 

Effect 

Signif. 

Level 

Marginal 

Effect 

Signif. 

Levei 

Total  CBT  Score: 

Commissioning  Source 
Only 

+.139 

.075 

+.153 

.037 

Full  Specification 

+.157 

.042 

+.169 

.016 

Maritime  Warfare 
Moduie: 

Commissioning  Source 
Only 

+..156 

.077 

+  .117 

.279 

Full  Specification 

+.181 

.089 

+.149 

.175 

DIVO  Fundamental 
Module: 

Commissioning  Source 
Only 

+.119 

.145 

+  .121 

.188 

Full  Specification 

+.146 

.078 

+.138 

.138 

Engineering  Moduie: 

Commissioning  Source 
Only 

+.258 

.111 

+.196 

.097 

Full  Specification 

+.265 

.000 

+.196 

.000 

Navigation- 
Seamanship  Moduie: 

Commissioning  Source 
Only 

+.023 

.717 

+.078 

.196 

Full  Specification 

NS 

NS 

Maritime  Warfare 

Exam: 

Commissioning  Source 
Only 

+.063 

.256 

+.057 

.246 

Full  Specification 

NS 

NS 

COVE  Exam: 

Commissioning  Source 
Only 

+.042 

.583 

-.056 

.246 

Full  Specification 

+.023 

.590 

-.112 

.050 

*Graduates  of  “most  selective”  colleges/universities  only. 
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The  first  noteworthy  outcome  is  that  NROTC  graduates  from  highly  selective 
schools  are  more  likely  to  pass  three  of  the  four  self-directed  modules  and  the 
overall  CBT  test  than  USNA  graduates.  The  passing  rate  differential  varies  between 
15%  to  25%  points  (as  compared  to  USNA  graduates).  These  estimated  marginal 
impacts  are  large,  significant,  and  robust  to  adding  control  variables  for  ship  type 
and  home  port,  academic  major,  demographics,  and  junior  officer  Navy  experience. 
We  also  find  that  the  few  OCS  graduates  of  “most  selective”  colleges  are  also  more 
likely  than  USNA  graduates  to  pass  the  overall  self-directed  CBT  test  score  and  the 
Engineering  module. 

These  results  are  surprising  for  two  reasons.  First,  Naval  Academy  graduates 
receive  the  most  Navy-specific  education  and  training  possible  and  have  greater 
technical  core  exposure  than  others  attending  most  selective  colleges.  One  possible 
explanation  for  the  outcomes  in  Table  5  is  that  performance  on  self-directed  DIVO 
training  modules — holding  all  other  observables  constant — mainly  reflects  self- 
motivation.  The  curriculum  at  USNA  is  highly  structured  with  compulsory  class 
attendance.  Grades  are  assigned  twice  during  the  semester  with  severe  penalties 
for  those  who  have  low  grades  or  are  failing  a  class,  especially  if  it  is  in  the  core 
technical  curriculum.  Rarely  do  USNA  midshipmen  find  academic  environments  that 
allow  students  to  determine  for  themselves  how  much  effort  they  exert  during  the 
semester.  Graduates  of  highly  selective  colleges  and  universities,  on  the  other  hand, 
are  far  more  likely  to  operate  in  learning  environments  that  give  greater  latitude  for 
student  self-motivation.  Other  than  flunking  out,  students  are  free  to  choose  how  to 
spend  their  time  and  effort  in  academic  pursuits — with  few,  if  any,  repercussions  for 
below-average  performance.  It  is  likely  that  the  learning  environment  for  the  self- 
directed  DIVO  training  modules  is  more  akin  to  that  experienced  by  graduates  of 
selective  civilian  colleges  and  universities  than  by  USNA  graduates. 

Another  set  of  findings  further  support  the  above  explanation  in  that  once 
officers  are  brought  together  in  the  ’’leveling  experience”  of  SWOSDOC,  few,  if  any, 
differences  in  performance  are  observed  among  students  from  different 
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commissioning  sources.  There  are  no  differences  in  passing  rates  of  the  more 
traditional  classroom  environment  for  the  Maritime  Warfare  course,  while  OCS 
graduates  are  11%  less  likely  than  USNA  graduates  to  pass  the  simulator  for  the 
COVE  course.  This  finding  may  support  the  notion  that  the  naval  seamanship  and 
navigation  courses  taken  by  all  midshipmen  give  them  an  edge  over  OCS 
graduates,  who  are  not  exposed  to  such  professional  academic  courses  during 
college. 

2.  College  Quality 

The  model  specifications  also  permit  one  to  estimate  the  marginal  effects 
uniquely  related  to  college  quality  among  NROTC  and  OCS  graduates.  As  noted 
above,  the  study  does  not  have  any  measure  of  academic  performance  of  individual 
officers,  so  the  quality  of  the  school  attended  is  used  as  a  proxy  for  “student  ability” 
in  an  academic  environment.  One  would  presume  that  graduates  of  higher  quality 
educational  institutions  would  be  more  likely  to  pass  these  self-directed  and 
classroom  tests  than  graduates  of  less  selective  colleges.. 

Table  8  below  lists  the  names  of  colleges  and  universities  attended  by  the 
officers  attending  SWOSDOC  Classes  213  through  224  during  2007  and  2008.  Over 
100  officers  attended  “Highly  Competitive”  schools — including  UC  Berkeley,  Lehigh, 
University  of  Miami,  and  large  competitive  state  schools  like  Michigan,  Minnesota, 
Penn  State,  and  Wisconsin.  Seventy  officers  attended  “Very  Competitive”  schools 
including  Virginia  Tech,  Purdue,  VMI,  and  Auburn.  The  largest  group  of  schools  (117 
officers)  attended  “Competitive  Colleges,”  including  Florida  State  and  the  University 
of  Florida,  Maryland,  Oklahoma,  and  The  Citadel.  A  small  number  of  officers  (19) 
attended  “Least  Competitive”  schools  like  Hampton  Roads,  Prairie  View  A&M,  and 
Southern  University. 
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Table  8.  List  of  Colleges  &  Universities  of  NROTC  &  OCS  Commissioned 
Officers  by  Barron’s  Quality  Index 


MOST  COMPETITIVE: 

HIGHLY  COMPETITIVE: 

VERY  COMPETITIVE: 

COMPETITIVE: 

LEAST  COMPETITIVE: 

Boston  College 

2 

Bentley  College 

1 

Auburn  Univ 

1 

Abilene  Christian  University 

1 

Hampton  Roads 

2 

Boston  University 

2 

UC  at  Berkeley 

3 

California  Poly  S  U 

1 

Arizona,  Univ  of 

10 

Idaho,  Univ 

7 

UCLA 

1 

Colorado  School  of  Mines 

1 

Drexel  Univ 

1 

California  Maritime  Academy 

1 

New  School  Univ 

1 

Carnegie  Mellon  Univ 

2 

Connecticut,  Univ  of 

1 

Duquesne  Univ 

2 

Central  Missouri  St  Univ 

1 

Norfolk  State  Univ 

2 

Connecticut  College 

1 

Fordham  Univ 

2 

Georgia  State  Univ 

3 

Colorado,  Univ  of 

5 

Norfolk  State  University 

1 

Cornell  Univ 

6 

George  Washington  Univ 

7 

Illinois,  Univ  of 

4 

Eastern  Michigan 

3 

Prairie  View  A  &  M  Univ 

3 

Georgetown  Univ 

2 

Georgia  Institute  of  Tech 

5 

Iowa  State 

3 

Embry-Riddle  Aero  Univ 

1 

Savannah  State  Univ 

1 

Harvard 

1 

Illinois  Institute  of  Tech 

1 

Louisiana  State  Univ 

2 

Florida  State  Univ 

2 

Southern  Univ 

1 

Holy  Cross,  College  of  the 

4 

Kettering  University 

1 

Loyola  Univ 

1 

Florida,  Univ  of 

5 

MIT 

1 

Lehigh 

1 

Missouri,  Univ  of 

5 

Hampton  University 

2 

New  York  Univ.  at  Buffalo 

1 

Marquette  Univ 

4 

Nebraska  Univ 

2 

Houston,  Univ  of 

1 

UNC  at  Chapel  Hill 

3 

Miami  of  Ohio 

3 

North  Florida,  Univ  of 

2 

Humboldt  State  Univ 

1 

Northeastern 

1 

Miami  Univ 

3 

Purdue  Univ 

3 

Jacksonville  Univ 

9 

Northwestern 

1 

Michigan,  Univ  of 

2 

Rochester  Inst  of  Tech 

4 

Liberty  Univ 

1 

Notre  Dame,  Univ  of 

11 

Minnesota  Univ 

1 

South  Carolina,  Univ  of 

3 

Maine  Maritime  Academy 

5 

Pennsylvania,  Univ  of 

4 

North  Carolina  State  Univ 

6 

South  Florida,  Univ  of 

6 

Maine  Univ 

2 

Rice  Univ 

1 

Ohio  State  Univ 

8 

St.  Thomas,  Univ  of 

1 

Maryland,  Univ  of 

4 

Rochester,  Univ  of 

3 

Penn  State 

13 

SUNY  Brockport 

1 

Massachusetts  Marit  Acad 

1 

Southern  California,  Univ  of 

5 

Pittsburgh,  Univ  of 

2 

SUNY  Stoybrook 

1 

Memphis  Univ 

3 

Tulane  Univ 

3 

Rensselaer  Poly  Institute 

2 

Utah  Univ 

2 

Mississippi,  Univ  of 

3 

University  of  Southern  Cal 

1 

San  Diego,  Univ  of 

11 

Virginia  Military  Institute 

3 

Morehouse  College 

1 

Vanderbilt  Univ 

5 

Syracuse  Univ 

1 

Virginia  Tech 

6 

New  Mexico,  Univ  of 

1 

Virginia,  University  of 

3 

Texas  A  &  M 

10 

Washington,  Univ  of 

9 

Norwich  Univ 

5 

Washington  and  Lee  Univ 

1 

Texas,  Univ  of 

2 

Worcester  Poly  Institute 

4 

Oklahoma  Univ 

4 

Villanova 

4 

Old  Dominion  Univ 

16 

Wisconsin-Madison  Univ 

4 

Oregon  State 

9 

Saint  Leo  University 

1 

San  Diego  State  Univ 

5 

San  Jose  State  Univ 

1 

Southern  Illinois  Univ 

1 

St.  Leo  College 

1 

Temple  Univ 

1 

The  Citadel 

8 

UMBC 

1 

Wayland  Baptist  Univ 

1 

Total 

65 

Total 

101 

Total 

70 

Total 

117 

Total 

19 

As  shown  in  Figure  15,,  the  passing  rate  on  the  weighted  total  self-directed 
CBT  test  varies  greatly  across  the  SWOSDOC  classes  by  college  quality.  In  general, 
the  passing  rate  tends  to  rise  over  time.  Often,  100%  of  graduates  of  the  most 
selective  colleges  achieve  passing  scores  on  the  weighted  self-directed  SWOSDOC 
test  score. 
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AVERAGE  PASSING  RATE  FOR  COMMISSIONED  OFFICERS 
BY  COLLEGE  QUALITY  (NROTC  &  OCS) 


Figure  15.  Passing  Rates  of  NROTC  &  OCS  Graduates  on  Totai  DIVO 
Fundamentals  Exam  by  College  Quality 

To  gain  a  better  understanding  of  the  possible  effect  of  school  quality  on 
passing  rates  on  DIVO  training  tests  and  exams,  we  once  again  structured  two- 
stage  econometric  models;  only  school  quality  is  included  in  the  first  model.  The 
second  model  includes  additional  explanatory  variables  related  to  test  performance. 
These  results  are  reported  in  Table  9. 

Since  one  would  expect  the  marginal  impacts  of  college  quality  to  be  negative 
(since  the  comparison  group  is  the  “most  selective”  schools),  and  the  size  of  the 
estimated  impacts  to  be  larger  the  less  competitive  the  school,  the  results  shown  in 
Table  9  are  surprising.  They  indicate  that  only  graduates  from  the  “highly  selective” 
(second  rank  in  quality)  and  the  “least  selective”  (lowest  rank  in  quality)  are  less 
likely  to  pass  most  of  the  self-directed  CBT  modules.  For  example,  graduates  of  the 
most  selective  and  least  selective  colleges  are  31%  less  likely  to  achieve  a  passing 
grade  on  the  combined  weighted  total  CBT  score  than  those  from  the  most  selective 
colleges.  Similar  findings  characterize  the  Maritime  Warfare  module.  It  is  also  of 
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interest  to  note  that  graduates  from  all  quality  levels — compared  to  the  omitted  “most 
selective”  category — are  far  less  likely  to  pass  the  Engineering  module,  with  lower 
passing  rates  varying  from  -25%  to  -42%.  Finally,  the  passing  rates  on  the  self- 
directed  CBT  tests  for  graduates  of  the  “very  selective”  colleges  rank  second  to 
those  from  “most  selective”  colleges.  The  only  passing  rates  that  are  significantly 
different  for  these  graduates  are  found  in  the  Engineering  module,  and  the  size  of 
the  marginal  impact  (-27.3%)  is  the  smallest  of  all  school  quality  categories  reported 
in  the  table. 
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Table  9.  Effects  of  College  Quality  on  Passing  Rates  on  DIVO 
Fundamental  Tests  and  Exams 


CBT  or  Classroom 
Exam 

Marginal  Effects  &  Significance  Levels  of  College  Quality* 

High 

Select 

ly 

:ive 

Ver 

Select 

y 

;ive 

Competitive 

Least 

Competitive 

M.E. 

Sig. 

Lev. 

M.E. 

Sig. 

Lev. 

M.E. 

Sig. 

Lev. 

M.E. 

Sig. 

Lev. 

Total  CBT  Score: 

School  Quality 

Only 

-.263 

.027 

-.123 

.241 

-.123 

.238 

-.261 

.024 

Full  Specification 

-.306 

.013 

-.164 

.136 

-.152 

.157 

-.303 

.011 

Maritime  Warfare 
Module: 

School  Quality 

Only 

-.273 

.000 

-.084 

.369 

-.113 

.222 

-.215 

.009 

Full  Specification 

-.300 

.000 

-.129 

.157 

-.149 

.099 

-.247 

.001 

DIVO  Fundamental 
Module: 

School  Quality 

Only 

-.087 

.414 

+.052 

.577 

-.222 

.815 

-.057 

.583 

Full  Specification 

-.138 

.213 

+.010 

.921 

-.047 

.636 

-.058 

.594 

Engineering 

Module: 

School  Quality 

Only 

-.424 

.002 

-.246 

.068 

-.317 

.015 

-.347 

.015 

Full  Specification 

-.451 

.002 

-.273 

.050 

-.346 

.010 

-.363 

.014 

Navigation- 

Seamanship 

Module: 

School  Quality 

Only 

-.060 

.517 

-.041 

.603 

-.027 

.726 

-.081 

.389 

Full  Specification 

NS 

NS 

NS 

NS 

Maritime  Warfare 
Exam: 

School  Quality 

Only 

+.034 

.608 

-.094 

.320 

-.101 

.275 

-.058 

.524 

Full  Specification 

NS 

NS 

NS 

NS 

COVE  Exam: 

School  Quality 

Only 

-.030 

.768 

-.016 

.863 

-.055 

.553 

-.056 

.579 

Full  Specification 

NS 

NS 

NS 

NS 

*Graduates  o 


NROTC  and  OCS 


only. 
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These  school  quality  differences,  however,  do  not  seem  to  affect  the  students’ 
performances  on  the  end-of-SWOSDOC  exams,  since  none  of  the  schooling 
variables  are  statistically  significant  predictors.  One  possible  explanation  for  this 
pattern  of  marginal  effects  could  be  that  school  quality  may  be  measuring 
differences  in  individual  motivation  levels  rather  than  differences  in  cognitive  skills.  It 
is  unfortunate  that  individual  college  grades  were  not  included  in  the  data  base,  as 
using  overall  school  quality  is  an  imperfect  proxy  for  individual  academic 
performance. 

C.  Academic  Majors 

The  academic  majors  of  commissioned  officers  were  categorized  into  six 
groups.  Thee  first  three  are  more  technical  in  nature  and  include  engineering, 
math/physical  sciences,  and  biological  sciences.  The  last  three  are  non-technical  in 
nature  and  include  business/economics,  the  social  sciences,  and  arts  and 
humanities.^®  The  passing  rates  on  the  weighted  total  CBT  tests  by  academic  majors 
are  shown  in  Figure  16  below.  This  chart  illustrates  that  engineers  generally  have 
higher  passing  rates  than  non-technical  majors. 


A  seventh  major  group — other  majors — is  included  in  the  statistical  models  but  has  only  eleven 
officers.  These  majors  are  so  different  from  the  others  that  they  were  not  included  with  the  others  and 
included  such  majors  as  physical  education,  design,  health  and  wellness,  and  human  nutrition. 
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AVERAGE  PASSING  RATE  FOR  COMMISSIONED  OFFICERS 
BY  ACADEMIC  MAJOR  GROUP 


Figure  16.  Passing  Rates  of  Commissioned  Officers  on  Total  DIVO 
Fundamentals  Exam  by  Academic  Majors 

To  gain  a  better  understanding  of  the  effect  of  academic  major  on  the  self- 
directed  CBT  tests  and  more  traditional  classroom  exams  at  SWOSDOC,  we  next 
developed  two  stage  models  of  the  probability  of  passing  these  tests.  In  these 
models,  only  college  major  is  specified  in  the  first  stage  and  other  significant 
explanatory  variables  are  included  in  the  second  stage.  The  results  of  these  logistic 
regression  models  are  displayed  in  Table  10  below. 

In  general,  there  is  little  difference  in  performance  on  the  self-directed  training 
modules  among  the  technical  majors.  The  only  exception  is  quite  surprising: 
math/physical  science  majors  are  17.5%  less  likely  to  pass  the  Engineering  module 
than  are  engineering  majors. 
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Table  10.  Effect  of  Academic  Major  on  Passing  Rates  on  DIVO 
Fundamental  Tests  and  Exams 


Marginal  Effects  (M.E.)  and  Significance  Levels  of  Academic  Majors 


CBT  or 

Math/Phys.Sci. 

Bio  Sciences 

Bus/Econ 

Social  Sciences 

Arts  & 
Humanities 

Other  Majors 

wIdoolUUlIl  CAalll 

M.E. 

Sig. 

Level 

M.E. 

Sig. 

Level 

M.E. 

Sig. 

Level 

M.E. 

Sig. 

Level 

M.E. 

Sig. 

Level 

M.E. 

Sig. 

Level 

Total  CBT  Score: 

Academic  Major  Only 

-.055 

.496 

-.129 

.178 

-.187 

.012 

-.102 

.113 

-.170 

.081 

-.188 

.259 

Full  Specification 

-.044 

.592 

-.087 

.369 

-.169 

.031 

-.095 

.154 

-.167 

.103 

-.134 

.438 

Maritime  Warfare 
Module: 

Academic  Major  Only 

-.063 

.386 

-.039 

.654 

-.092 

.165 

+.009 

.880 

-.042 

.629 

-.096 

.515 

Full  Specification 

NS 

NS 

NS 

NS 

NS 

NS 

DIVO  Fundamental 
Module: 

Academic  Major  Only 

-.064 

.413 

-.044 

.632 

-.094 

.185 

-.079 

.210 

-.179 

.047 

-.029 

.858 

Full  Specification 

-.074 

.362 

-.060 

.533 

-.113 

.130 

-.111 

.095 

-.178 

.057 

-.083 

.627 

Engineering  Module: 

Academic  Major  Only 

-.162 

.068 

-.061 

.531 

-.214 

.009 

-.137 

.040 

-.274 

.010 

+.040 

.792 

Full  Specification 

-.175 

.053 

-.051 

.591 

-.215 

.009 

-.163 

.017 

-.280 

.010 

+.027 

.857 

Navigation- 
Seamanship  Module: 

Academic  Major  Only 

-.007 

.906 

-.038 

.625 

-.065 

.295 

-.045 

.388 

-.047 

.557 

+.063 

.560 

Full  Specification 

NS 

NS 

NS 

NS 

NS 

NS 

Maritime  Warfare 

Exam: 

Academic  Major  Only 

-.013 

.847 

-.210 

.070 

-.234 

.010 

-.109 

.078 

-.041 

.641 

* 

Full  Specification 

-.007 

.910 

-.228 

.057 

-.230 

.011 

-.111 

.069 

-.048 

.596 

* 

COVE  Exam: 

Academic  Major  Only 

+.034 

.621 

+.017 

.831 

-.167 

.020 

-.076 

.204 

-.123 

.187 

-.092 

.563 

Full  Specification 

+.040 

.553 

+.026 

.743 

-.182 

.016 

-.119 

.061 

-.160 

.109 

-.211 

.247 

*  All  eleven  officers  in  this  “other  majors”  category  passed,  which  forces  the  variable  to  be  excluded  from  the  logistic 
regression. 


One  expected  outcome  of  this  research  is  the  finding  that  officers  with  non¬ 
technical  majors  are  less  likely  to  pass  the  self-directed  training  modules  than  are 
engineering  majors.  For  example,  commissioned  officers  with  business/economics 
or  arts  and  humanities  majors  are  17%  less  likely  to  score  a  passing  grade  on  the 
combined  weighted  CBT  test  compared  to  engineers,  and  both  are  less  likely  to 
pass  the  engineering  module  than  engineers  (-22%  and  -28%,  respectively).  In 
addition,  social  science  majors  are  less  likely  to  pass  the  DIVO  Fundamentals  and 
the  Engineering  modules  than  engineers  (-11%  and-16%,  respectively).  In  general, 
these  findings  suggest  that  many  of  the  competencies  required  to  pass  the  self- 
directed  DIVO  training  modules  are  more  quantitative  and/or  technical  in  nature  and 
that  academic  major  plays  an  important  role  in  predicting  achievement  on  these  self- 
directed  tests. 

The  impact  of  college  major  also  correlates  to  performance  in  the  more 
traditional  classroom  training  programs  at  SWOSDOC.  Business/economics  majors 
are  23%  less  likely  to  pass  the  Maritime  Warfare  exam  and  18%  less  likely  to  pass 
the  COVE  simulator  test  than  are  engineering  majors.  Likewise,  social  science 
majors  are  11%  less  likely  to  pass  the  Maritime  Warfare  exam  and  12%  less  likely  to 
pass  the  COVE  simulator  test  than  are  engineering  majors.  In  summary,  knowledge 
acquired  in  technical  majors  appears  to  be  related  to  performance  on  both  self- 
directed  test  onboard  ships  as  well  as  exams  administered  in  the  classroom 
environment  of  SWOSDOC. 

D.  Demographic  Factors 

The  last  two  factors  included  in  the  statistical  models  of  self-directed  CBT 
tests  and  SWOSDOC  exams  are  ethnicity  and  gender.  Ethnicity  plays  essentially  no 
role  in  explaining  the  performance  on  the  self-directed  DIVO  training  modules  or  the 
traditional  classroom  Maritime  Warfare  course  given  at  SWOSDOC. It  does. 


”  The  only  exception  is  found  with  African-American  commissioned  officers,  who  are  16%  less  likely 
to  pass  the  self-directed  DIVO  Fundamentals  module  than  are  White  majority  officers. 
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however,  play  a  significant  role  in  explaining  the  passing  rate  on  the  simulator 
environment  COVE  exam.  As  shown  in  Table  11  below,  all  ethnic  minorities  are  less 
likely  to  pass  the  COVE  exam  than  are  members  of  the  White  majority.  Specifically, 
African-Americans  and  Asian  Pacific  Islanders  are  roughly  25%  less  likely  to  pass 
this  exam,  and  Hispanics  are  34%  less  likely  to  pass  the  COVE  exam  than  are 
Caucasian  officers.  Clearly,  more  analysis  is  required  to  gain  a  full  understanding  of 
these  ethnic  differences  that  are  strong  and  significant,  as  there  is  no  evidence  of 
such  differences  for  all  other  self-directed  or  traditional  classroom  training  programs. 

Table  11.  Effect  of  Race  and  Ethnicity  on  Passing  Rates  on 
SWOSDOC  COVE  Exam 


COVE  Exam 

Marginai  Effects  &  Significance 
Levels  of  Ethnicity 

Ethnicity 

Only 

Full  Model 
Specification 

Marg. 

Effect 

Sig. 

Level 

Marg. 

Effect 

Sig. 

Levei 

African  American 

-.240 

.006 

-.246 

.009 

Hispanic 

-.267 

.015 

-.339 

.003 

Asian  Pacific  Isiands 

-.245 

.010 

-.265 

.007 

Other  Ethnic 

-.219 

.028 

-.216 

.034 

1.  Gender 

As  noted  in  Figure  17,  the  passing  rate  on  the  weighted  total  CBT  score  for 
females  is  generally  below  that  of  males.  This  result  is  unexpected,  as  typically 
females  test  either  the  same  or  better  than  males  in  most  administered  tests.  To 
better  understand  this  outcome,  we  constructed  two-stage  logistic  models  on  the 
probability  of  passing  self-directed  tests  and  classroom  training  exams.  In  the  first 
stage,  only  gender  is  included  as  an  explanatory  variable;  in  the  second  stage,  other 
significant  factors  related  to  test  performance  are  added  so  that  we  could  derive  a 


PRAtST^NTIA  PER  SCISNTIA\t 


MANPOWER,  PERSONNEL,  TRAINING  &  EDUCATION  RESEARCH 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


-84- 


better  estimate  of  the  independent  and  unique  impact  gender  may  have  on  testing 
performance.  The  results  of  these  logistic  regression  models  are  displayed  in  Table 
12. 


Quite  surprisingly,  female  officers  are  less  likely  to  pass  two  of  the  four  self- 
directed  DIVO  training  models,  as  well  as  to  score  below  the  weighted  total  CBT 
score.  These  differences  in  passing  rates  are  large  and  significant — with  females 
being  29%  less  likely  to  pass  the  overall  CBT  test,  27%  less  likely  to  pass  the 
Maritime  Warfare  module,  and  16%  less  likely  to  pass  the  Engineering  module 
compared  to  male  officers.  It  may  also  be  noted  that  these  marginal  effects  are 
larger  than  in  the  restricted  model  that  includes  only  gender — suggesting  that  control 
factors  on  average  favor  females  over  males.  For  example,  females  are  more  likely 
to  be  assigned  to  a  CRUDES  ship  (58%  vs.  49%)  and  to  be  assigned  to  the  West 
Coast  (47%  vs.  34%) — which,  as  noted  earlier,  is  correlated  with  higher  passing 
rates  on  the  self-directed  training  modules. 

It  is  also  of  interest  to  note  that  gender  differences  in  passing  rates  are  not 
found  with  the  SWOSDOC  classroom  program  for  Maritime  Warfare.  The  marginal 
effect  for  females  is  positive  but  not  statistically  significant  compared  with  a  highly 
negative  and  significant  estimate  for  the  self-directed  Maritime  Warfare  module.  In 
addition,  no  significant  gender  differences  are  found  in  passing  rates  on  the  COVE 
exam.  These  findings  suggest  that  the  environment  aboard  ships  on  which  self- 
directed  learning  occurs  may  be  quite  different  for  female  compared  with  male 
officers.  Many  background  variables — such  as  commissioning  source,  school  quality 
and  academic  majors — are  controlled,  as  are  the  ship  type  and  home  port 
assignments.  In  spite  of  these  controls,  achievement  on  self-directed  tests  by 
females  is  lower  than  for  males  but  is  essentially  the  same  in  the  SWOSDOC 
training  program.  These  findings  may  suggest  a  need  for  improved  mentoring  of 
female  junior  officers,  who  may  easily  get  forgotten  in  the  hectic  life  aboard  ships. 
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VII 


Summary  and  Recommendations 


The  general  success  of  self-directed  CBT  paired  with  OJT  is  found  to  vary 
significantly  with  respect  to  ship  type  and  home  port  assignment,  as  well  as  with 
respect  to  pre-commissioning  factors  such  as  commissioning  source,  quality  of 
undergraduate  college,  and  academic  majors,  as  well  as  demographic 
characteristics,  especially  gender.  Considerations  for  rewarding  exemplary 
achievement  on  these  tests  are  also  noted. 

A.  Ship  Type  and  Home  Port  Assignments 

Officers  assigned  to  non-CRUDES  ships  are  significantly  less  likely  to  pass 
the  self-directed  CBT  modules  aboard  ships.  At  this  point,  one  can  only  conjecture 
why  this  is  so.  It  is  possible  that  the  hierarchical  command  structure  of  amphibious 
ships,  which  includes  many  non-surface  warfare  officers,  and  the  size  of  aircraft 
carriers  accounts  for  much  of  why  junior  officers  from  these  ship  types  fail  to  do  as 
well  as  those  on  CRUDES  ships.  If  the  Surface  Warfare  community  is  serious  about 
improving  JO  training,  we  recommend  that  more  resources  be  directed  to  determine 
the  reasons  for  this  anomaly.  Similarly,  we  recommend  that  future  research  be 
conducted  on  why  junior  officers  with  home  ports  on  the  Atlantic  coast  (i.e.,  Norfolk 
and  Jacksonville)  have  significantly  lower  passing  rates  on  Division  Officer 
Fundamental  CBT  tests  than  others.  Researchers  must  distinguish  possible 
differences  due  to  OPTEMPO  schedules  from  cultural  differences  that  may  exist  in 
the  senior  commands  between  Pacific  and  Atlantic  fleet  ports.  To  the  extent  that 
these  differences  in  self-directed  training  mirror  larger  differences  in  junior  officer 
training  opportunities,  we  recommend  that  the  surface  community  consider 
developing  incentives  that  will  alter  the  motivation  of  both  junior  officers  and  their 
immediate  superiors  regarding  the  time  devoted  to  and  the  value  of  self-directed 
CBT. 
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B.  Pre-commissioning  Factors 

The  study  found  significant  differences  in  the  success  of  self-directed  CBT 
aboard  ships  across  all  the  commissioning  programs.  Most  surprisingly,  Naval 
Academy  graduates  had  significantly  lower  passing  rates  than  otherwise  similar 
officers  commissioned  via  NROTC  and  OCS.  Given  the  numerous  explanatory 
variables  included  in  the  models,  the  study  concludes  that  observed  differences  in 
passing  are  most  likely  related  to  differences  in  motivation  that  result  from  the 
structure  and  rigor  in  their  academic  programs.  It  is  quite  possible  that  observed 
differences  in  self-directed  CBT  tests  aboard  ships  are  due,  to  a  large  extent,  to  how 
motivated  individuals  are  to  do  well  on  the  CBT.  Without  any  formal  initial  classroom 
training,  all  Division  Officers  must  find  time  in  a  very  busy  day  to  study,  generally  on 
their  own  time.  One  possibility  is  that  graduates  of  the  more  selective  civilian 
colleges  are  already  highly  self-motivated  to  study  on  a  regular  basis  without  direct 
supervision — perhaps  more  so  than  Naval  Academy  midshipmen.  On  the  other 
hand,  USNA  midshipmen  have  been  exposed  to  far  greater  levels  of  military 
indoctrination  than  graduates  of  other  sources,  which  would  rule  out  their  being  ill- 
prepared  for  the  material  covered  in  the  four  self-directed  training  modules.  We 
recommend  that  future  research  should  investigate  this  difference. 

The  study  also  found  that  the  quality  of  a  student’s  undergraduate  college — 
used  as  a  proxy  for  general  academic  ability — is  directly  related  to  performance  on 
the  self-directed  CBT  modules.  Graduates  of  the  most  selective  colleges,  including 
USNA,  have  significantly  higher  passing  rates  than  those  from  less  selective 
schools.  We  recommend  the  Navy  consider  both  directing  more  resources  to 
students  enrolling  in  NROTC  units  at  more  prestigious  schools,  as  well  as  directing 
greater  efforts  on  part  of  OCS  recruiters  towards  the  more  selective  colleges. 

The  study  also  finds  that  graduates  with  technical  majors  have  higher  passing 
rates  on  the  self-directed  CBT  tests  than  those  with  non-technical  skills.  We  realize 
we  must  be  careful  here  in  drawing  conclusions,  since  it  may  be  possible  that  much 
of  the  skill  content  of  the  self-directed  training  modules  may  be  more  quantitative 
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than  qualitative  in  nature.  Nowhere  are  Division  Officers  tested  on  inter-personal 
skills  that  may  be  directly  related  to  leadership  or  military  career  aspirations.  Clearily, 
the  self-directed  training  modules  only  capture  a  portion  of  the  skills  need  by  JOs  to 
be  successful  DIVOs.  Nonetheless,  it  is  still  important  for  senior  leadership  aboard 
ships  to  point  out  to  new  Division  Officers  this  finding  regarding  technical  knowledge 
and  to  ensure  that  more  time  be  devoted  towards  learning  from  these  self-directed 
training  modules  for  those  without  technical  degrees.  In  addition,  we  recommend 
that  the  SWO  community  ensure  that  the  material  covered  in  these  four  modules  be 
integrated  into  the  curriculum  taken  near  graduation  for  midshipmen  from  USNA  and 
NROTC. 

C.  Demographic  Factors 

One  of  the  most  important  findings  of  the  study  is  that  females  have 
significantly  lower  passing  rates  on  the  self-directed  CBT  modules,  but  comparable 
pass  rates  in  the  SWOSDOC  classroom  exams.  Given  the  many  control  factors 
specified  in  the  model,  the  most  likely  explanation  for  this  finding  once  again  may 
well  be  motivational  differences  by  gender.  For  example,  female  officers  are 
relatively  scarce  aboard  naval  ships  of  all  types,  which  may  make  it  difficult  to  find 
strong  mentors  who  can  provide  sound  professional  advice,  while  being  fully  upright 
and  truthful  regarding  their  performance  relative  to  male  junior  officers.  We 
recommend  that  senior  leadership  aboard  ships  be  made  aware  of  the  gender 
performance  differential  and  ensure  that  more  effort  on  the  part  of  female  Division 
Officers  be  directed  to  the  self-directed  training.  It  is  important  to  note,  moreover, 
that  the  same  can  be  said  regarding  all  aspects  of  the  culture  and  environment  of 
junior  officer  training  in  general.  One  cannot  presume  that  female  DIVOs  experience 
the  same  training  environment  as  do  their  male  counterparts,  and  the  researchers 
highly  recommended  that  greater  resources  be  directed  toward  a  better 
understanding  of  gender  differences  that  may  exist  in  the  early  developmental 
training  aboard  ships  of  junior  SWOs. 
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In  general,  the  study  finds  few  differences  in  self-directed  CBT  tests  by 
ethnicity.  The  only  exception  to  this  outcome  is  on  the  Conning  Officer  Virtual 
Environment  (COVE)  exam  following  one-on-one  experience  in  a  simulation 
environment  that  is  given  toward  the  end  of  SWOSDOC.  The  study  finds  that 
minorities  are  far  less  likely  to  pass  this  exam,  which  appears  to  be  composed  of 
fundamental  skills  relevant  to  passing  the  Surface  Warfare  Qualification  on  their 
ships.  Future  researchers  need  to  analyze  these  exam  results  in  more  detail  to  see 
where  and  why  minorities  are  performing  below  their  counterparts.  We  recommend 
that  SWOS  consider  extending  time  at  SWOSDOC  to  anyone  not  passing  this 
important  exam  so  that  all  who  complete  the  “leveling  experience”  can  be  counted 
as  having  passed  this  exam. 

In  summary,  the  most  important  results  of  the  study  suggest  that  the  success 
of  training  opportunities  for  Division  Officers  differs  significantly  between  self- 
directed  CBT  aboard  ships  and  the  more  traditional  classroom  environment  of 
SWOSDOC.  The  time  directed  to  and  motivation  to  acquire  self-directed  CBT  varies 
widely  across  ship  types,  home  ports  and  several  demographic  characteristics — 
including  gender,  commissioning  source,  college  quality  and  academic  major  of 
newly  commissioned  JOs.  Most,  if  not  all,  of  these  differences  seem  to  disappear 
when  junior  officers  are  given  introductory  training  in  a  traditional  classroom 
environment  in  which  DIVOs  are  given  the  single  task  of  learning  new  information 
without  other  day-to-day  responsibilities  aboard  ships.  The  instruction  is  highly 
standardized  and  communicated  by  instructors  whose  sole  task  is  to  teach  and 
instruct  junior  officers.  This  study,  if  nothing  else,  underscores  the  realization  that 
currently  little  is  known  regarding  the  impact  of  ship  type  and  home  port  on  training 
opportunities,  let  alone  why  differences  in  learning  occur  due  to  gender  or 
commissioning  source.  It  is  highly  recommended  that  additional  resources  be 
directed  to  a  better  understanding  of  learning  and  training  opportunities  aboard  ships 
before  final  decisions  are  made  regarding  the  division  of  resources  between 
introductory  classroom  and  early  CBT  aboard  ships.  It  is  important  that  the  Surface 
Community  both  standardizes  its  introductory  training  of  junior  officers  as  well  as 
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ensures  the  initial  training  is  highly  effective  and  conducive  to  learning  what  is 
required  to  be  a  successful  career  surface  warfare  officer. 


D.  Rewarding  Exemplary  Behavior 

As  noted  above,  one  way  to  improve  motivation  to  complete  the  self-directed 
training  is  to  adopt  proper  incentives  for  both  Division  Officers  and  their  senior 
leadership  aboard  ships.  Leadership  at  SWOSDOC  has  already  begun  such  efforts 
by  making  the  results  of  the  initial  test  scores  transparent  to  ship  commands  and  by 
rewarding  those  who  achieve  superior  scores  on  three  selected  tests  and  exams. 

The  first  two  include  the  CBT  modules  of  Maritime  Warfare  (score  of  84  or 
higher  with  a  range  of  42-100)  and  Division  Officer  Fundamentals  (score  of  84  or 
higher  with  a  range  of  44-100).  The  third  is  the  Rules-of-the-Road  exam  at 
SWOSDOC  (score  of  95  or  higher  with  a  range  of  68-102).’^  Out  of  the  733  Division 
Officers  who  completed  the  SWOSDOC  program  during  the  period  covered  by  this 
study,  30  (or  4%)  achieved  superior  scores  according  to  the  incentive  program. 

Table  13  below  indicates  the  distribution  of  officers  with  superior  scores  by  ship  type 
and  home  port  assignments.  Those  assigned  to  destroyers  or  cruisers  are  twice  as 
likely  to  earn  this  prestigious  reward  as  those  on  frigates,  who  were  three  times  as 
likely  to  earn  the  reward  as  those  assigned  to  amphibious  ships,  carriers,  and  small 
or  auxiliary  ships.  In  addition,  those  assigned  to  the  Western  Pacific  and  Japan  are 
twice  as  likely  to  earn  the  reward  as  those  from  Atlantic  home  ports,  and  six  times  as 
likely  as  those  from  Gulf  shore  ports.  If  the  surface  warfare  community  wants  to 
provide  incentives  to  all  its  junior  officers,  we  recommend  that  more  be  done  to 
standardize  junior  officer  training  opportunities  across  ship  types  and  home  port 
assignments,  or  to  develop  different  awards  for  those  with  significantly  different  initial 
assignments. 


The  ranges  noted  in  parentheses  are  those  reflected  from  the  sample  of  Division  Officers  in  the 
analysis  data  base  of  the  study. 
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Table  13.  Number  and  Percent  Achieving 
Superior  Performance 


Division  Officers  Achieving  Su 

perior  Performance 

on  Surface  Warfare  Fundameni 

tal  Tests  and  Exams 

Ship  Type; 

Number 

Total 

Percent 

Destroyer 

16 

258 

6.2 

Cruiser 

7 

104 

6.7 

Frigates 

3 

122 

2.5 

Amphibious 

4 

187 

2.1 

Small 

Boat/Auxiliary 

0 

38 

0 

Carriers 

0 

24 

0 

30 

733 

4.1 

Flome  Port: 

Atlantic 

10 

358 

2.8 

Western  Pacific 

16 

279 

5.7 

Japan 

4 

68 

5.9 

Other 

0 

28 

0 

TOTAL 

30 

733 

4.1 

In  summary,  the  study  addresses  personal  achievement  on  initial  junior  officer 
tests  and  exams,  which  are  related  to  the  technical  skills  and  competencies  of 
Division  Officers.  While  uncovering  achievement  differences  on  these  tests  that  are 
related  to  pre-commissioning  factors,  the  study  finds  additional  variations  in  test 
performance  related  to  the  environments  aboard  ships  and  across  home  ports  that 
may  possibly  be  indicative  of  a  larger  issue  related  to  the  early  training  opportunities 
of  junior  officers  in  the  surface  Navy.  To  the  extent  this  is  true,  it  is  in  the  best 
interest  of  the  community  to  investigate  in  more  depth  why  training  environments  for 
junior  officers  differ  to  the  extent  that  has  been  found  in  this  study. 
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